rng0.io https://rng0.io jondot. Rust, tools, building. Zola en Sat, 24 Oct 2026 00:00:00 +0000 Trust Your Own Hash, Not the Publisher's Sat, 24 Oct 2026 00:00:00 +0000 Unknown https://rng0.io/blog/trust-your-own-hash-not-the-publishers/ https://rng0.io/blog/trust-your-own-hash-not-the-publishers/ <p>When the attacker controls the publisher, everything the publisher signs is the attacker's.</p> <p>On October 19, according to OVHcloud, someone used a developer's GitHub account to <a href="https://www.it-connect.tech/ovhcloud-cli-binaires-on-github-were-tampered-with-by-an-infostealer/">swap the OVHcloud CLI release binaries</a> for v0.11.0 through v0.15.0 with an infostealer. For v0.15.0 they also rewrote <code>checksums.txt</code>, so if you did the responsible thing and checked your download against the release page, it matched. The swap was live for 94 minutes, and it was caught because <strong>a customer's CI noticed that the SHA-256 of the Linux arm64 archive had changed</strong>.</p> <p>Two days later, <a href="https://docs.litellm.ai/blog/security-incident-oct-21-2026">LiteLLM 1.107.0</a> went out to PyPI and to Docker Hub and GHCR, including <code>latest</code>. Stolen developer GitHub credentials were used to point the release workflow at a lookalike repo, <code>Berri-AI/litellm</code>, and a second developer's credential passed the production approval gate. The project's own report says the official <code>v1.107.0</code> source tag doesn't contain the malicious files. The real pipeline built, signed and shipped the attacker's code. And the payload sits in a <code>.pth</code> file, so, as <a href="https://www.microsoft.com/en-us/security/blog/2026/10/23/inside-the-litellm-1-107-0-supply-chain-compromise-credential-theft-through-python-startup-hooks/">Microsoft's write-up</a> puts it, it runs at Python startup "without requiring the application to actively use LiteLLM gateway functionality."</p> <p>Same shape, different month: in August the <a href="https://www.aikido.dev/blog/keyv-and-friends-compromised-in-npm-supply-chain-attack">keyv packages</a> went out from a hijacked maintainer GitHub account, <em>with valid provenance signed by GitHub Actions</em>. And on October 5 a modified <a href="https://www.stepsecurity.io/blog/subql-ecosystem-compromised">SubQuery publish workflow</a> swapped the <code>@subql/common</code> sources for a tarball from an outside domain right before publishing, with, in StepSecurity's words, "no hash or signature verification of that external archive in the added step."</p> <h2 id="the-publisher-is-the-attack-surface">The Publisher Is the Attack Surface</h2> <p>Provenance, trusted publishing and signed releases are good. They answer <em>where</em> an artifact was built and <em>who</em> published it. Trusted publishing with token publishing turned off kills a whole class of attacks -- a stolen npm token used from some random laptop, for example.</p> <p>But look at what these attacks did instead. They didn't go around the publisher, <strong>they became the publisher</strong>. They took the accounts and the workflows, and from there every publisher-side signal said yes: the checksums file, the provenance, the signature, the <code>latest</code> tag.</p> <p>I wrote about this in 2021 when we released <a href="/blog/stopping-supply-chain-attacks-with-preflightspectral/">Preflight</a>: the chain of trust behind a <code>curl | sh</code> has too many links to verify, so move it to the one link you control, a digest that is "produced and maintained by you, and only you." Five years later, that's still the check an attacker with the maintainer's GitHub account can't touch. And there are two of them you can own: the hash, and the wait.</p> <h2 id="mutable-things-pin-the-hash-yourself">Mutable Things: Pin the Hash Yourself</h2> <p>GitHub release assets and container tags can change under the same name. That's how OVHcloud's binaries were swapped in place, and how a floating <code>latest</code> pointed at a bad LiteLLM image. For these, record the hash <em>at the time you vetted the thing</em>, in your own repo, and check against that:</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> expected.sha256 holds the bare hex digest, recorded by you when you vetted this release</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-support z-function z-echo z-shell">echo</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span><span class="z-meta z-group z-expansion z-command z-parens z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-parens z-begin z-shell">(</span><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">cat</span></span><span class="z-meta z-function-call z-arguments z-shell"> expected.sha256</span><span class="z-punctuation z-section z-parens z-end z-shell">)</span></span> ovhcloud-cli.tar.gz<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span></span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sha256sum</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>check</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>strict</span> -</span> </span></code></pre> <p>If someone replaces the archive, this fails and your pipeline stops, regardless of what the release page says. This is exactly what <a href="https://github.com/spectralops/preflight">Preflight</a> does for scripts and executables (<code>preflight run sha256=...</code>), and it's the kind of check that caught the OVHcloud swap.</p> <p>For images, the same idea is a digest instead of a tag: deploy <code>image@sha256:...</code>, not <code>image:latest</code>. Microsoft's advice after LiteLLM says the same thing -- hash-pinned lock files, and container digests instead of floating tags.</p> <p>For packages, your lockfile already does it. <code>npm ci</code> installs exactly what <code>package-lock.json</code> says and checks each tarball against the integrity hash stored there. On the Python side:</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">uv</span></span><span class="z-meta z-function-call z-arguments z-shell"> pip compile requirements.in<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>generate-hashes</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>o</span> requirements.txt</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">pip</span></span><span class="z-meta z-function-call z-arguments z-shell"> install<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>require-hashes</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>r</span> requirements.txt</span> </span></code></pre> <p>The first line resolves and writes a hash next to every package; the second refuses to install anything that doesn't match.</p> <h2 id="new-versions-wait-a-day">New Versions: Wait a Day</h2> <p>Hash pinning protects you from a version that <em>changes</em>. LiteLLM 1.107.0 and the keyv releases didn't change -- they were <strong>new versions</strong>. Bump your lockfile at the wrong hour and you'll pin the malware, with a perfectly matching hash.</p> <p>The fix for that is old advice: time proven over just released. I wrote that in 2015 in <a href="/blog/technological-inflation/">Technological Inflation</a> about adopting frameworks, and it applies to patch releases just as well. The LiteLLM wheels were on PyPI from 11:39 to 13:23 UTC, and the bad Docker Hub tags were gone by 14:20. A one-day delay on new versions sits that out.</p> <p>Package managers support this now. In pnpm (10.16 and up), in <code>pnpm-workspace.yaml</code>:</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">minimumReleaseAge</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-constant z-numeric z-integer z-decimal z-yaml">1440</span> <span class="z-comment z-line z-number-sign z-yaml"><span class="z-punctuation z-definition z-comment z-line z-number-sign z-yaml">#</span> minutes; only install versions at least a day old </span></span></code></pre> <p>The <a href="https://pnpm.io/settings/dependency-resolution#minimumreleaseage">pnpm docs</a> list 1440 as the default from v11. In uv, in <code>pyproject.toml</code>:</p> <pre data-lang="toml" class="language-toml z-code"><code class="language-toml" data-lang="toml"><span class="z-source z-toml"><span class="z-punctuation z-definition z-table z-begin z-toml">[</span><span class="z-meta z-tag z-table z-toml"><span class="z-entity z-name z-table z-toml">tool</span><span class="z-punctuation z-separator z-table z-toml">.</span><span class="z-entity z-name z-table z-toml">uv</span></span><span class="z-punctuation z-definition z-table z-end z-toml">]</span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">exclude-newer</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>1 day<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span> </span></code></pre> <p><a href="https://docs.astral.sh/uv/reference/settings/">uv accepts</a> friendly durations like <code>24 hours</code> or <code>1 week</code>, as well as timestamps. Pick a longer window if you like; the point is that the delay is yours, not the publisher's.</p> <h2 id="the-tradeoff">The Tradeoff</h2> <p>A cooldown also delays the fix you actually want. When a real security patch lands, you'll need to let it through on purpose -- pnpm has <code>minimumReleaseAgeExclude</code> for that. That's fine. <strong>An exception you made deliberately, for one package, is the whole point</strong>: someone looked.</p> <p>And yes, if everyone waits a day, someone still has to install first. That's what the scanners are for. <a href="https://socket.dev/blog/litellm-compromise">Socket</a> had malware alerts on the LiteLLM wheels at 11:46 UTC, seven minutes after they were published. Your day of waiting is what lets that work reach you before the package does.</p> <p>None of this replaces what publishers are doing: immutable releases, provenance, signing. OVHcloud shipped v0.16.0 as an immutable release with a signed tag, which is the right call. But that protects their side. On your side, the only hash that counts is the one you recorded.</p> You Can Only Rotate the Secrets You Can Find Sat, 24 Oct 2026 00:00:00 +0000 Unknown https://rng0.io/blog/you-can-only-rotate-the-secrets-you-can-find/ https://rng0.io/blog/you-can-only-rotate-the-secrets-you-can-find/ <p>"Rotate your keys" is the standard advice after an incident. It assumes you know what all of your keys are. Usually you don't.</p> <p><a href="https://zeabur.com/blogs/october-2026-security-incident">Zeabur's incident report</a> is unusually candid about this. After an incident in August they rotated keys, and the August report said "all internal admin keys have been disabled or rotated." Their October report calls that statement incomplete, and says why: "The August rotation did not cover the cluster admin tokens and AWS keys stored in plaintext in the configuration repository..." Those admin tokens were created in January 2024 with no expiry, and in January 2026 one was written in plaintext to a private configuration repository. On October 14 the attacker used a Zeabur AWS key and then a cluster admin token. Among the exposed data: 3,218,965 service environment variable records.</p> <p><strong>The secrets you don't know about are the ones that stay valid.</strong></p> <h2 id="deleted-is-not-gone">Deleted Is Not Gone</h2> <p>The second place forgotten secrets live is Git history. <a href="https://thehackernews.com/2026/10/credential-stealing-github-actions.html">GhostAction</a>, a campaign first seen in September 2025, is back. It pushes a fake "security audit" workflow into repos, most likely using leaked maintainer tokens. Among other things it scans "the entire git history" to find, per StepSecurity as quoted by The Hacker News, "credentials that may have inadvertently committed to the repository and subsequently deleted." Socket counts more than 500 GitHub accounts pushing the workflow to tens of thousands of repositories since October 7.</p> <p>So a key that someone committed, noticed, and removed in a follow-up commit is still in the repo. It's in every clone and every fork, and the attacker's script looks for it. You can see this with nothing but Git:</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> every commit reachable from any ref that added or removed something shaped like an AWS access key ID</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">git</span></span><span class="z-meta z-function-call z-arguments z-shell"> log<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>all</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>G</span> <span class="z-string z-quoted z-single z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&#39;</span>AKIA[0-9A-Z]{16}<span class="z-punctuation z-definition z-string z-end z-shell">&#39;</span></span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>oneline</span></span> </span></code></pre> <p><code>-G</code> matches the regex against each commit's diff, so the commit that removed the key shows up next to the one that added it. A tool like <a href="https://github.com/gitleaks/gitleaks">gitleaks</a> does the same across a large set of secret patterns, and it reads history by running <code>git log -p</code> under the hood.</p> <p><strong>When a secret has been in a commit, deleting it is not remediation -- revoking it is.</strong> Rewriting history is cleanup; it doesn't make the key invalid.</p> <h2 id="build-the-list-before-you-need-it">Build the List Before You Need It</h2> <p>I wrote about secret sprawl in 2021, in <a href="/blog/last-mile-of-sensitive-datasolved-with-teller/">the Teller post</a>: "Often secret sprawl happens without individuals realizing it, because it’s naturally hard to see the bigger picture..." That's still the problem. You can't fix sprawl in the hours after an incident, because that's when you're working from memory.</p> <p>What works is boring, and it happens before the incident:</p> <ul> <li><strong>Keep secrets in a store, not in files.</strong> Config repos, <code>.env</code> files and CI variables get copied around, and every copy is one more place to search. A reference to a vault entry isn't the secret, and the vault knows every secret it holds.</li> <li><strong>Give everything an expiry.</strong> A token with no end date stays valid until someone remembers it, and a token that expires cleans up after the rotations you miss.</li> <li><strong>Scan history, not just HEAD.</strong> Run the scan on full history once, and revoke whatever it finds, including keys that were removed years ago.</li> <li><strong>Make rotation a script.</strong> If rotating everything takes a script, you can run it. If it takes a meeting, that's where things get missed.</li> </ul> <p>The cost is real: a vault is one more service to run, expiring tokens break things at inconvenient times, and the first full-history scan of a big monorepo can take a while. That's still cheaper than learning about your inventory from an attacker.</p> <p>Credit where it's due: Zeabur published a detailed timeline that includes what went wrong, and that's more useful to everyone else than a vague statement. Read it, then go find the tokens you forgot about.</p> Your Fuzz Harness Is the Spec Sat, 24 Oct 2026 00:00:00 +0000 Unknown https://rng0.io/blog/your-fuzz-harness-is-the-spec/ https://rng0.io/blog/your-fuzz-harness-is-the-spec/ <p>Anthropic just <a href="https://www.anthropic.com/research/formal-verification">proved that 239 OSS-Fuzz entry points don't crash on any input under 64 MB</a>, and <strong>every spec was a fuzz test somebody had already written.</strong></p> <p>Here's what Anthropic and Theorem did. They took 239 entry points across 123 open-source libraries from Google's OSS-Fuzz, turned each existing fuzz test into a statement -- "this never crashes" -- over the <em>compiled</em> RISC-V binary, and had Claude write a proof that the Rocq proof checker accepts. The claim: for every input under 64 MB, the program exits normally, with no memory-safety errors and no failed assertions. Along the way Claude found hundreds of bugs, patched them, and re-proved. All of this is Anthropic's own reporting; nobody outside has replicated it yet.</p> <p>The proofs are enormous (about 180 million lines of Rocq in total) and no human has to read them -- the checker does. <strong>What a human has to read is the spec</strong>, and here the spec was a fuzz harness: a few lines of code that take arbitrary bytes and call your library with them.</p> <h2 id="what-stays-expensive">What Stays Expensive</h2> <p>Whatever an agent does with your code -- fuzz it, refactor it, prove it -- it needs to be told what "correct" means. That part doesn't get automated away.</p> <p>A fuzz harness is the most boring way to say it. No oracle, no expected output, just "for any input, don't fall over." It's also the one a machine can do the most with: a fuzzer can run it for a week, and as the researchers just showed, a proof can be built on it.</p> <p>I've long made fixture and snapshot tests a big part of how I test, and I've argued for <a href="/blog/why-you-should-canary-test-in-production/">canary testing in production</a>. Same reason every time: tests that keep paying after you write them. A fuzz harness is the extreme case -- a few lines, and it was enough of a spec to hang a machine-checked proof on.</p> <h2 id="write-one-today">Write One Today</h2> <p>In Rust, that's <a href="https://github.com/rust-fuzz/cargo-fuzz">cargo-fuzz</a>. <code>cargo install cargo-fuzz</code>, then <code>cargo fuzz init</code> in your crate, and a target like this:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#!</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">no_main</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-keyword z-other z-rust">use</span> <span class="z-meta z-path z-rust">libfuzzer_sys<span class="z-punctuation z-accessor z-rust">::</span></span>fuzz_target<span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"> </span><span class="z-source z-rust"><span class="z-support z-macro z-rust">fuzz_target!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">|</span></span></span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-variable z-parameter z-rust">data</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>[<span class="z-storage z-type z-rust">u8</span>]<span class="z-punctuation z-section z-parameters z-end z-rust">|</span></span> </span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-control z-rust">if</span> <span class="z-storage z-type z-rust">let</span> <span class="z-support z-type z-rust">Ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>s</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">std<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">str<span class="z-punctuation z-accessor z-rust">::</span></span>from_utf8<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>data</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">let</span> <span class="z-keyword z-operator z-rust">_</span> <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">my_crate<span class="z-punctuation z-accessor z-rust">::</span></span>parse<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>s</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <p>That's the whole spec: for any UTF-8 string, <code>parse</code> returns without panicking. It doesn't care whether the result is <code>Ok</code> or <code>Err</code>, only that you get one. Run it with <code>cargo +nightly fuzz run &lt;target&gt;</code> (it needs nightly for some unstable flags).</p> <p>In safe Rust this mostly finds panics -- an <code>unwrap</code> on bad input, an out-of-bounds index, overflow in debug builds -- not memory corruption. That's still a crash in production. Then pick the functions that eat untrusted input -- parsers, decoders, anything behind a network port -- and give each one a harness. That's your list of specs.</p> <h2 id="where-this-stops">Where This Stops</h2> <p>Anthropic lists the limits itself, and they matter:</p> <ul> <li><strong>"Formally verified" does not mean "bug-free."</strong> If the spec is wrong or too weak, you have a proof of the wrong thing. "Doesn't crash" says nothing about whether <code>parse</code> returns the right numbers.</li> <li><strong>The machine isn't yours.</strong> The proofs assume a static, single-threaded, musl-linked RISC-V binary with a fixed memory budget. Most people ship dynamically linked glibc on x86 or ARM. And none of the verified code here was Rust.</li> <li><strong>It covers what it covers.</strong> Only the verified entry points, memory safety as far as the tooling detects it, no dead code, and nothing about callers misusing the API.</li> <li><strong>It's expensive.</strong> Anthropic says the proofs can take hundreds of millions of tokens.</li> </ul> <p>Don't wait for an agent to prove your service. If this ever becomes routine, the first projects to benefit are the ones that already wrote down what "correct" means, in code -- and most of the time, that means a fuzz harness.</p> Reviving a 6-Year-Old Rust Project Sat, 06 Apr 2024 00:00:00 +0000 Unknown https://rng0.io/blog/reviving-a-six-year-old-rust-project/ https://rng0.io/blog/reviving-a-six-year-old-rust-project/ <h2 id="introduction">Introduction</h2> <p><img src="https://rng0.io/blog/reviving-a-six-year-old-rust-project/1.webp" alt="" /></p> <p>In this post I'll be reviving <a href="https://github.com/jondot/rawsort">a Rust project</a> that has been dormant for six years! I'll share the challenges and insights gained while upgrading this project to a modern Rust edition. Whether you're a Rustacean dealing with legacy code or just curious about the evolution of Rust, this post is for you.</p> <p>The project itself is called <a href="https://github.com/jondot/rawsort">rawsort</a> and it's one of the first CLIs I built in Rust that I've used daily. It watches for my camera memory drive being mounted, and then auto imports, sorts, and shapes my image catalog automatically.</p> <h2 id="moving-from-pre-2018-rust-to-rust-2021-edition">Moving from pre-2018 Rust to Rust 2021 Edition</h2> <p>Six years in the programming world is akin to a few decades in other fields. Rust has undergone significant changes in this period. The language has become more mature, with enhancements in safety, performance, and usability. These improvements, while beneficial, introduce a gap between legacy and current code.</p> <p>First thing to do, is upgrade into the Rust 2021 edition:</p> <pre data-lang="toml" class="language-toml z-code"><code class="language-toml" data-lang="toml"><span class="z-source z-toml"><span class="z-comment z-line z-number-sign z-toml"><span class="z-punctuation z-definition z-comment z-toml">#</span> cargo.toml</span> </span><span class="z-source z-toml"> </span><span class="z-source z-toml"><span class="z-punctuation z-definition z-table z-begin z-toml">[</span><span class="z-meta z-tag z-table z-toml"><span class="z-entity z-name z-table z-toml">package</span></span><span class="z-punctuation z-definition z-table z-end z-toml">]</span> </span><span class="z-source z-toml"><span class="z-comment z-line z-number-sign z-toml"><span class="z-punctuation z-definition z-comment z-toml">#</span> add this</span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">edition</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>2021<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span> </span><span class="z-source z-toml"> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">name</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>rawsort<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">version</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>0.4.1-alpha.0<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">authors</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-punctuation z-definition z-array z-begin z-toml">[</span><span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>Dotan Nahum &lt;jondotan@gmail.com&gt;<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span><span class="z-punctuation z-definition z-array z-end z-toml">]</span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">publish</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-constant z-language z-toml">false</span> </span></code></pre> <h2 id="handling-edition-errors">Handling edition errors</h2> <p>First thing to do, is let's drop <code>extern crate</code> instructions. Clippy tells us exactly about this, and offers to fix it as well.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-module z-rust">mod</span> <span class="z-entity z-name z-module z-rust">executor</span><span class="z-punctuation z-terminator z-rust">;</span></span> </span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-storage z-type z-module z-rust">mod</span> <span class="z-entity z-name z-module z-rust">exif</span><span class="z-punctuation z-terminator z-rust">;</span></span> </span><span class="z-source z-rust"><span class="z-keyword z-other z-rust">extern</span> <span class="z-keyword z-other z-rust">crate</span> chrono<span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-keyword z-other z-rust">extern</span> <span class="z-keyword z-other z-rust">crate</span> clap<span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-keyword z-other z-rust">extern</span> <span class="z-keyword z-other z-rust">crate</span> dialoguer<span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-keyword z-other z-rust">extern</span> <span class="z-keyword z-other z-rust">crate</span> exif <span class="z-keyword z-operator z-rust">as</span> ext_exif<span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-keyword z-other z-rust">extern</span> <span class="z-keyword z-other z-rust">crate</span> itertools<span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-keyword z-other z-rust">extern</span> <span class="z-keyword z-other z-rust">crate</span> walkdir<span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-module z-rust">mod</span> <span class="z-entity z-name z-module z-rust">registry</span><span class="z-punctuation z-terminator z-rust">;</span></span> </span></code></pre> <p>into</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-module z-rust">mod</span> <span class="z-entity z-name z-module z-rust">executor</span><span class="z-punctuation z-terminator z-rust">;</span></span> </span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-storage z-type z-module z-rust">mod</span> <span class="z-entity z-name z-module z-rust">exif</span><span class="z-punctuation z-terminator z-rust">;</span></span> </span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-module z-rust">mod</span> <span class="z-entity z-name z-module z-rust">registry</span><span class="z-punctuation z-terminator z-rust">;</span></span> </span></code></pre> <h2 id="handling-actual-code-errors">Handling actual code errors</h2> <p>Correcting for the evolution of <code>dyn</code>:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">error[E0782]: trait objects must include the `dyn` keyword </span><span class="z-text z-plain"> --&gt; src/registry.rs:6:39 </span><span class="z-text z-plain"> | </span><span class="z-text z-plain">6 | map: HashMap&lt;String, (String, Box&lt;Fn(&amp;exif::Exif, &amp;DirEntry) -&gt; String&gt;)&gt;, </span><span class="z-text z-plain"> | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ </span><span class="z-text z-plain"> | </span><span class="z-text z-plain">help: add `dyn` keyword before this trait </span><span class="z-text z-plain"> | </span><span class="z-text z-plain">6 | map: HashMap&lt;String, (String, Box&lt;dyn Fn(&amp;exif::Exif, &amp;DirEntry) -&gt; String&gt;)&gt;, </span><span class="z-text z-plain"> | +++ </span></code></pre> <p>And now we're going to solve for the removed <code>extern</code>s:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">error[E0412]: cannot find type `Exif` in crate `exif` </span><span class="z-text z-plain"> --&gt; src/registry.rs:17:22 </span><span class="z-text z-plain"> | </span><span class="z-text z-plain">17 | T: Fn(&amp;exif::Exif, &amp;DirEntry) -&gt; String + &#39;static, </span><span class="z-text z-plain"> | ^^^^ not found in `exif` </span><span class="z-text z-plain"> | </span><span class="z-text z-plain">help: consider importing this struct </span><span class="z-text z-plain"> | </span><span class="z-text z-plain">1 + use crate::exif::Exif; </span><span class="z-text z-plain"> | </span><span class="z-text z-plain">help: if you import `Exif`, refer to it directly </span><span class="z-text z-plain"> | </span><span class="z-text z-plain">17 - T: Fn(&amp;exif::Exif, &amp;DirEntry) -&gt; String + &#39;static, </span><span class="z-text z-plain">17 + T: Fn(&amp;Exif, &amp;DirEntry) -&gt; String + &#39;static, </span><span class="z-text z-plain"> | </span></code></pre> <p>Fixing another <code>use</code> statement:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> before </span></span><span class="z-source z-rust"><span class="z-keyword z-other z-rust">use</span> registry </span><span class="z-source z-rust"> </span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> after </span></span><span class="z-source z-rust"><span class="z-keyword z-other z-rust">use</span> <span class="z-keyword z-other z-rust">crate</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>registry </span></code></pre> <p>Fixing another <code>use</code> rename import issue:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> before </span></span><span class="z-source z-rust"><span class="z-keyword z-other z-rust">use</span> <span class="z-meta z-path z-rust">ext_exif<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span>Reader<span class="z-punctuation z-separator z-rust">,</span> Tag</span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"> </span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> after </span></span><span class="z-source z-rust"><span class="z-keyword z-other z-rust">use</span> <span class="z-meta z-path z-rust">exif<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span>Reader<span class="z-punctuation z-separator z-rust">,</span> Tag</span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <p>Here's a prehistoric finding!</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">error: use of deprecated `try` macro </span><span class="z-text z-plain"> --&gt; src/main.rs:224:43 </span><span class="z-text z-plain"> | </span><span class="z-text z-plain">224 | let mut watcher: RecommendedWatcher = try!(Watcher::new(tx, Duration::from_secs(2))); </span><span class="z-text z-plain"> | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ </span><span class="z-text z-plain"> | </span><span class="z-text z-plain"> = note: in the 2018 edition `try` is a reserved keyword, and the `try!()` macro is deprecated </span><span class="z-text z-plain">help: you can use the `?` operator instead </span><span class="z-text z-plain"> | </span><span class="z-text z-plain">224 - let mut watcher: RecommendedWatcher = try!(Watcher::new(tx, Duration::from_secs(2))); </span><span class="z-text z-plain">224 + let mut watcher: RecommendedWatcher = Watcher::new(tx, Duration::from_secs(2))?; </span><span class="z-text z-plain"> | </span><span class="z-text z-plain">help: alternatively, you can still access the deprecated `try!()` macro using the &quot;raw identifier&quot; syntax </span><span class="z-text z-plain"> | </span><span class="z-text z-plain">224 | let mut watcher: RecommendedWatcher = r#try!(Watcher::new(tx, Duration::from_secs(2))); </span><span class="z-text z-plain"> | ++ </span></code></pre> <p>Happily refactoring the <code>try!</code> instruction into a <code>?</code> unwrapping works flawlessly.</p> <p><strong>That's it</strong> other than a few warnings due to some improvements in Clippy suggesting some neater and smarter ways to write Rust, the code is now upgraded.</p> <h2 id="handling-library-breakages">Handling library breakages</h2> <p>There is still one last error that stems from the <code>slog</code> logging library. Since we upgraded from an old Rust edition, and <code>use</code> and macros semantics changed, we cannot use its macros from the new <code>2021</code> edition of Rust.</p> <p>We'll upgrade <code>slog</code>:</p> <pre data-lang="toml" class="language-toml z-code"><code class="language-toml" data-lang="toml"><span class="z-source z-toml"><span class="z-comment z-line z-number-sign z-toml"><span class="z-punctuation z-definition z-comment z-toml">#</span> latest version</span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">slog</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-punctuation z-definition z-inline-table z-begin z-toml">{</span> <span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">version</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>2.7.0<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span> <span class="z-punctuation z-definition z-inline-table z-end z-toml">}</span> </span></code></pre> <p>This resolves all remaning issues and we can use macros in a more modern way:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> the macros `error`, `info`, `o` </span></span><span class="z-source z-rust"><span class="z-keyword z-other z-rust">use</span> <span class="z-meta z-path z-rust">slog<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span>error<span class="z-punctuation z-separator z-rust">,</span> info<span class="z-punctuation z-separator z-rust">,</span> o<span class="z-punctuation z-separator z-rust">,</span> Drain<span class="z-punctuation z-separator z-rust">,</span> Fuse<span class="z-punctuation z-separator z-rust">,</span> Logger</span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <h2 id="we-re-done">We're done!</h2> <p><img src="https://rng0.io/blog/reviving-a-six-year-old-rust-project/2.webp" alt="" /></p> <p>The whole process took less than 10 minutes. And we have a fully-working Rust project, building into a 6 years newer stack!</p> <p>I challenge you to find any other programming langauge, make upgrades and have this kind of smooth sailing. In terms of programming languages, we just did magic!</p> <p>Add to that the fact that the Rust authors from six years ago cloud never have predicted what Rust in 2024 will be and so could never imaging how an upgrade experience will be. But, Rust is encouraging a <em>safe</em> and <em>stable</em> mindset all the way from its inception, ensured a safe future for upgrades.</p> <h2 id="future-proofing-the-project">Future-Proofing the Project</h2> <p>Keeping a project up-to-date is a continual effort. But Rust proves that even if you have a very old codebase, you shouldn't be discouraged.</p> <p>There's very little to be learned here for "future proofing" your code, as most of the changes came from the new Rust editions, however we can have some notes in mind:</p> <ul> <li>Breakage in codebase between major language upgrades is <em>guarded by editions</em> and this really works, it's not just theory</li> <li>When a language and ecosystem starts out there's always a "race to logging APIs", and so logging libraries tend to historically break their interfaces a lot</li> <li>There might be a nice place for an "edition tester" CI job, where you can run a codebase on a matrix of different editions, and decide when the amount of upgrades calls for an action</li> </ul> <h2 id="conclusion">Conclusion</h2> <p>This upgrade journey has been <em>easy</em>. Rust's evolution over these years has been impressive, and adapting to these changes is a safe, encouraging and pain free experience. This only proves yet again, if you want to build software, build it in Rust. If you're an open source maitainer which cares for their time, energy and sanity -- this recommendation is 10x more important. Rust gives you the freedom to not worry about your existing open source project portfolio and not to get buried in maintenance hell.</p> Introducing Loco: the "Rust on Rails" Sat, 24 Feb 2024 00:00:00 +0000 Unknown https://rng0.io/blog/introducing-loco/ https://rng0.io/blog/introducing-loco/ <h2 id="introduction-to-loco">Introduction to Loco</h2> <p><img src="https://rng0.io/blog/introducing-loco/1.webp" alt="" /></p> <p><a href="https://loco.rs/">Loco is a Web or API framework</a> for Rust: a "<strong>Rust on Rails</strong>". Strongly inspired by Rails, it contains everything you need to go from side project to a startup in Rust.</p> <p>Loco is built by <a href="https://twitter.com/jondot">@jondot</a> and <a href="https://github.com/kaplanelad">@kaplanelad</a> together with the <a href="https://github.com/loco-rs/loco/graphs/contributors">Loco contributors</a>.</p> <h3 id="the-genesis-of-loco-bridging-the-gap-in-web-development">The Genesis of Loco: Bridging the Gap in Web Development</h3> <p>In order to build Loco, we had to develop our set of: <strong>principles, guides, philosophies and opinions</strong> to answer:</p> <ul> <li>Where should we make a difference?</li> <li>How can we fully take advantage of Rust's abilities?</li> <li>How can we make hard things simple, and impossible things possible?</li> </ul> <p>Believe it or not, Loco <strong>is a 3rd rewrite of building a Rails alternative</strong> which is not based on Ruby.</p> <p>The first iteration was originally built on Node.js, a stack of high-productivity libraries glued together with some special sauce, made building a SaaS for a startup a breeze. It became: <a href="https://hyperstackjs.io/">hyperstackjs</a>, a "Rails on Typescript".</p> <p>But <strong>Node.js and the Javascript/Typescript ecosystem has too much mental fatigue</strong>. I could not handle the churn, the ecosystem. Some times it took as little as 4 months to go back to a project and find that it needed so many upgrades for dependencies, and everything broke.</p> <p><img src="https://rng0.io/blog/introducing-loco/3.webp" alt="" /></p> <p>So how about a crazy idea: <strong>can this be built in Rust?</strong></p> <p>Crazy because Rust is rigid, static, safe. On the surface traits that are supposed to be opposite to what Rails and Ruby are: free, easy, flexible, productive.</p> <p>You'd be surprised to know that Rust is <strong>perfect</strong> for this job. The first port of <em>Hyperstack</em> to Rust was called <strong>RustyRails</strong> and then changed into <strong>Loco</strong>.</p> <p>Today, all the boxes were checked. I have zero mental overhead, zero churn of dependencies, zero fatigue working with Rust and Loco.</p> <p>Oh, and <strong>I got 100,000req/s back with zero effort</strong>. This is Rust.</p> <h2 id="overview-of-loco-core-features-and-philosophies">Overview of Loco: Core Features and Philosophies</h2> <p>A framework is all about balances. Loco <strong>creates balance over these principles</strong>:</p> <ul> <li><strong>Safe, robust software</strong> powered by Rust</li> <li>That can be <strong>played with</strong>, experimented with, iterated on rapidly</li> <li><strong>Offers everything Rails has</strong> which includes: data access, controllers, views, background jobs, websockets, mailers, storage, and even things that are offered from Rails gems such as authentication, i18n, pagination -- all built in.</li> <li>Has an easy and <strong>simple ergonomics</strong> and API surface area for developers.</li> </ul> <h3 id="why-rust-understanding-the-language-behind-loco">Why Rust? Understanding the Language Behind Loco</h3> <p>First off, if you love Ruby, and happy with it -- stop here and go with Rails. Rails is fantastic, and there is <strong>nothing</strong> that can compare to it.</p> <p>However, if Ruby is not something you use or can use, you most probably have 3 valid and popular options:</p> <ul> <li>Node.js</li> <li>Go</li> <li>Java/JVM</li> </ul> <p><strong>None of these have a proper Rails-like framework</strong>. Node.js has <strong>some options</strong> which can function as a partially implemented Rails in terms of features (e.g. Adonis and others).</p> <p>But <strong>Node.js has Javascript fatigue</strong>, which, I have to say, in 2024 is only becoming exponentially worse and not better.</p> <p><strong>Rust</strong> can be more expressive and much more safe than Go, and can be more performant, simpler, requires less tweaks and quirks and lighter on systems than the JVM.</p> <p>It is the perfect language for developer happiness, <strong>zero effort performance</strong>, simple and robust software that is easy to operate in production.</p> <h3 id="language-differences-rust-vs-ruby">Language Differences: Rust vs. Ruby</h3> <p><img src="https://rng0.io/blog/introducing-loco/2.webp" alt="" /></p> <p>You may be coming from Ruby or Rust, but not both. If that is the case,here's a quick comparison between the two to get some initial perspective.</p> <h5 id="purpose-and-design-philosophy">Purpose and Design Philosophy</h5> <ul> <li><strong>Rust</strong>: Systems programming, emphasizing performance and safety.</li> <li><strong>Ruby</strong>: Emphasizes simplicity and productivity, often used for web development.</li> </ul> <h5 id="memory-management">Memory Management</h5> <ul> <li><strong>Rust</strong>: Ownership model, no garbage collector.</li> <li><strong>Ruby</strong>: Garbage-collected, simpler but with performance overhead.</li> </ul> <h5 id="type-system">Type System</h5> <ul> <li><strong>Rust</strong>: Strong, static type system with type inference.</li> <li><strong>Ruby</strong>: Dynamic typing, flexible but prone to runtime errors.</li> </ul> <h5 id="performance">Performance</h5> <ul> <li><strong>Rust</strong>: High performance, comparable to C/C++.</li> <li><strong>Ruby</strong>: Generally slower due to being interpreted.</li> </ul> <h5 id="concurrency">Concurrency</h5> <ul> <li><strong>Rust</strong>: Advanced concurrency support, avoids data races.</li> <li><strong>Ruby</strong>: Supports concurrency, but hindered by Global Interpreter Lock (GIL).</li> </ul> <h5 id="community-and-ecosystem">Community and Ecosystem</h5> <ul> <li><strong>Rust</strong>: Growing, with a focus on safe concurrency. Expanding Cargo and Crates.io ecosystem.</li> <li><strong>Ruby</strong>: Large, established community. Rich library of gems, especially for web development.</li> </ul> <h2 id="the-modern-web-developer-s-toolkit-what-loco-brings-to-the-table">The Modern Web Developer’s Toolkit: What Loco Brings to the Table</h2> <h3 id="simplicity">Simplicity</h3> <p><strong>Rust dependencies are stable, simple, usually non-breaking</strong> because of the awesome Rust community. And then, there is just one tool for packages, linting, building and so on, and all tools are powered by Cargo.</p> <p><strong>With Loco you build just a single binary</strong> - that's your app. Deploy is just as easy as copying your binary to your server.</p> <h3 id="safety-performance">Safety &amp; Performance</h3> <p>Loco is not a small framework <strong>but also not a big framework</strong>. We managed to do as little work as possible and then hand off to a time tested, stable and mature libraries to do the grunt work.</p> <p>This recipe makes Loco <strong>super fast</strong>. On a typical M1 macbook a benchmark which includes database access is in the area of 30,000req/s.</p> <h3 id="concurrency-made-simple">Concurrency Made Simple</h3> <p><strong>We run on Tokio and Axum</strong> on the server, and on Sidekiq-rs for background workers. We support a transparent and powerful concurrency model which can be switched:</p> <ol> <li><strong>Async in-process</strong>, evented: powered by Tokio. Just run tokio tasks as you wish.</li> <li><strong>Async background process</strong>: enqueue a task and get it performed on the background on the same machine or different machine. Uses a Redis queue under the hood.</li> </ol> <p>You can switch the model <strong>from configuration</strong>.</p> <h3 id="what-can-5-buy-me">What can $5 buy me?</h3> <p>Almost all architectural decision were made by building a <strong>"one person framework"</strong>. That is a framework which lets you be hyper-productive, easy on your brain, but also easy on your budget.</p> <p>For example, you can get a free hosting provider with <strong>very low</strong> resource numbers, and deploy your Loco app because a Loco app is just a 20mb binary, and using Rust means it is very light on memory and CPU.</p> <p>You can have background jobs for free if you run async in tokio. Or you can spend $5 for a small Redis instance and have distributed background jobs.</p> <p><strong>Everything in Loco is optimized to get you started quick and cheap</strong>, on your own. You don't need a team, an architect, a specific cloud provider a budget or anything else in order to start other than your passion for your new project!</p> <h2 id="key-features-of-loco">Key Features of Loco</h2> <p><img src="https://rng0.io/blog/introducing-loco/4.webp" alt="" /></p> <h3 id="batteries-included-or-lean-mean-choose-any">Batteries included or Lean &amp; mean: choose any</h3> <p><strong>One of the key mindsets of Rust is to reason about cost</strong>. This is why building a "Rails on Rust" has to have knobs for developers to turn off anything that they consider too "heavy" or too "costly".</p> <p>All around Loco, <strong>you can switch off parts of the framework</strong> in compile time using Rust features, and create your own learn, lightweight apps. For example:</p> <ul> <li>You can switch off the ORM/data stack completely, or pick specific databases to compile in</li> <li>You can remove any layer of the "MVC" model by simply deleting files (you pay no cost for not using those)</li> <li>You can switch off the authentication flows, websockets, CLI, storage providers and more</li> </ul> <p>In fact, if you go through the starters wizard, you get something similar to "Loco editions" where each starter contains various hand-selected "batteries" to include.</p> <h3 id="rust-rails-ergonomics">Rust &amp; Rails ergonomics</h3> <p><strong>We do everything in Rust</strong>. If you imagine building Rails: a flexible, maleable, sometime DSL-looking framework, in Rust -- then Rust has a lot of missing pieces to provide that fantastic "playdough-like" DSL code.</p> <p><strong>One example can be dynamic loading of modules like Rails has</strong>. In Rails when you drop a file in a folder, it is picked up and becomes live on the next app restart. Instead of fighting and providing dynamic code loading for Rust, which will be foreign to Rust developers, we decided to avoid doing that completely, which means, if you have a new controller to add to your app, <strong>you specify it the Rust way: explicitly</strong>.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> specifying routes and controllers in src/app.rs </span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust">Hooks <span class="z-keyword z-other z-rust">for</span></span><span class="z-meta z-impl z-rust"> <span class="z-entity z-name z-impl z-rust">App</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> .. </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">routes</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">_ctx</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>AppContext</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> AppRoutes</span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-path z-rust">AppRoutes<span class="z-punctuation z-accessor z-rust">::</span></span>with_default_routes<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">add_route</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">controllers<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">notes<span class="z-punctuation z-accessor z-rust">::</span></span>routes<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">add_route</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">controllers<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">user<span class="z-punctuation z-accessor z-rust">::</span></span>routes<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span>.. </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>We try to <strong>balance magic and Rust-isms</strong> in Loco. We work hard on ergonomics, and when in doubt we always do what a Rust programmer would expect and not what a Ruby on Rails developer would expect.</p> <h3 id="intuitive-design-and-ease-of-use">Intuitive Design and Ease of Use</h3> <p>Loco sticks to the MVC (Model-View-Controller) abstraction for Web frameworks. Present in many paradigm shifts since the 90's, MVC have been able to reduce complexity for developers. This is what Rails does, too.</p> <p>This means you can expect a similar intuitive folder layout like Rails has:</p> <table><thead><tr><th>File/Folder</th><th>Purpose</th></tr></thead><tbody> <tr><td><code>src/</code></td><td>Contains controllers, models, views, tasks and more</td></tr> <tr><td><code>app.rs</code></td><td>Main component registration point. Wire the important bits here.</td></tr> <tr><td><code>lib.rs</code></td><td>Various rust-specific exports of your components.</td></tr> <tr><td><code>bin/</code></td><td>Has your <code>main.rs</code> file, you don't need to worry about it</td></tr> <tr><td><code>controllers/</code></td><td>Contains controllers, all controllers are exported via <code>mod.rs</code></td></tr> <tr><td><code>models/</code></td><td>Contains models, <code>models/_entities</code> contains auto-generated SeaORM models, and <code>models/*.rs</code> contains your model extension logic, which are exported via <code>mod.rs</code></td></tr> <tr><td><code>views/</code></td><td>Contains JSON-based views. Structs which can <code>serde</code> and output as JSON through the API.</td></tr> <tr><td><code>workers/</code></td><td>Has your background workers.</td></tr> <tr><td><code>mailers/</code></td><td>Mailer logic and templates, for sending emails.</td></tr> <tr><td><code>fixtures/</code></td><td>Contains data and automatic fixture loading logic.</td></tr> <tr><td><code>tasks/</code></td><td>Contains your day to day business-oriented tasks such as sending emails, producing business reports, db maintenance, etc.</td></tr> <tr><td><code>tests/</code></td><td>Your app-wide tests: models, requests, etc.</td></tr> <tr><td><code>config/</code></td><td>A stage-based configuration folder: development, test, production</td></tr> <tr><td><code>channels/</code></td><td>Contains all channels routes.</td></tr> </tbody></table> <h3 id="rapid-development-and-deployment">Rapid Development and Deployment</h3> <p>We drive rapid development with <strong><code>cargo loco generate</code></strong>:</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo loco generate scaffold movie title:string</span> </span></code></pre> <p>Which is similar to Rails' generators, and has the same motivation: to get you started quickly. The code <code>loco generate</code> creates is not just demo code, it is safe Rust. You can keep it and build on it.</p> <p>We also support <strong><code>cargo loco deploy</code></strong> which will offer multiple ways to deploy, and once you select your deployment method, will generate all the necessary files and configuration for your deploy as well as modify your app if needed. This is something <strong>Rails does not have</strong> out of the box.</p> <h3 id="saas-authentication-built-in">SaaS authentication built in</h3> <p>Where usually with Rails you had to pick and specify a library like Devise, <strong>we provide the entire auth stack included with Loco</strong>. You can get both JWT or API key based authentication, which is modern, safe, and with security best-pracices baked-in.</p> <p>The authentication flow is end-to-end and fully customizable, from registration to verification emails and more.</p> <h3 id="test-first-design">Test first design</h3> <p><strong>Every component is fully testable</strong>. Rust being a static and safe programming language, achieving this key principle is much more complex than any other dynamic language, where you can monkey-patch, inject, or dynamically change code.</p> <p>That said, Loco provides all needed facilities to make testing a breeze. From a convenient and ergonomic test kit for custom tailored for each app layer (models, views, workers, etc.) to snapshot testing to save on typing big robotic right-left assertion blocks at the end of each test.</p> <h3 id="a-single-binary-to-rule-them-all">A single binary to rule them all</h3> <p>We hold the principle that <strong>your Loco app is a single binary</strong>. Everything is embedded (other than server-side templates and assets which are held on disk on purpose since it's a different use case and are an optional feature): your code, workers, even email templates. You can deploy it to the cloud by having a Docker have just a single binary, or copy it manually to a Raspberry Pi. Having a single binary is magical in a way that it enables a diverse set of use cases.</p> <h2 id="getting-started-with-loco">Getting Started with Loco</h2> <p><img src="https://rng0.io/blog/introducing-loco/7.webp" alt="" /></p> <h3 id="setting-up-your-development-environment">Setting Up Your Development Environment</h3> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo install loco-cli</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo install sea-orm-cli</span> </span></code></pre> <p>If you need, you can run Postgres and Redis (for background jobs) via Docker. Note <code>myapp_development</code> is specific to an app called <code>myapp</code> which we will soon choose:</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> docker run<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>d</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>p</span> 5432:5432<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>e</span> POSTGRES_USER=loco<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>e</span> POSTGRES_DB=myapp_development<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>e</span> POSTGRES_PASSWORD=<span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>loco<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span> postgres:15.3-alpine</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> docker run<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>p</span> 6379:6379<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>d</span> redis redis-server</span> </span></code></pre> <h3 id="creating-your-first-loco-project">Creating Your First Loco Project</h3> <p>Run <code>loco new</code> (the <code>loco</code> binary is provided by <code>loco-cli</code>):</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> loco new</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">✔</span></span><span class="z-meta z-function-call z-arguments z-shell"> ❯ App name<span class="z-keyword z-operator z-regexp z-quantifier z-shell">?</span> · myapp</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">?</span></span><span class="z-meta z-function-call z-arguments z-shell"> ❯ What would you like to build<span class="z-keyword z-operator z-regexp z-quantifier z-shell">?</span> ›</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">lightweight-service</span></span><span class="z-meta z-function-call z-arguments z-shell"> (minimal, only controllers and views</span><span class="z-meta z-function-call z-shell"></span>) </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Rest</span></span><span class="z-meta z-function-call z-arguments z-shell"> API (with DB and user auth</span><span class="z-meta z-function-call z-shell"></span>) </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">❯</span></span><span class="z-meta z-function-call z-arguments z-shell"> Saas app (with DB and user auth</span><span class="z-meta z-function-call z-shell"></span>) </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">🚂</span></span><span class="z-meta z-function-call z-arguments z-shell"> Loco app generated successfully in:</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">myapp</span></span> </span></code></pre> <p>See that everything is OK with the <code>doctor</code> command:</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cd myapp</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo loco doctor</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo loco doctor</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Finished</span></span><span class="z-meta z-function-call z-arguments z-shell"> dev <span class="z-keyword z-control z-regexp z-set z-begin z-shell">[</span>unoptimized + debuginfo<span class="z-keyword z-control z-regexp z-set z-end z-shell">]</span> target(s</span><span class="z-meta z-function-call z-shell"></span>) <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">in</span></span><span class="z-meta z-function-call z-arguments z-shell"> 0.32s</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Running</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-meta z-group z-expansion z-command z-backticks z-shell"><span class="z-punctuation z-section z-group z-begin z-shell">`</span><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">target/debug/myapp-cli</span></span><span class="z-meta z-function-call z-arguments z-shell"> doctor</span><span class="z-punctuation z-section z-group z-end z-shell">`</span></span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">✅</span></span><span class="z-meta z-function-call z-arguments z-shell"> SeaORM CLI is installed</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">✅</span></span><span class="z-meta z-function-call z-arguments z-shell"> DB connection: success</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">✅</span></span><span class="z-meta z-function-call z-arguments z-shell"> Redis connection: success</span> </span></code></pre> <p>Finally, start your engines!</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo loco start</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">▄</span></span><span class="z-meta z-function-call z-arguments z-shell"> ▀</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">▀</span></span><span class="z-meta z-function-call z-arguments z-shell"> ▄</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">▄</span></span><span class="z-meta z-function-call z-arguments z-shell"> ▀ ▄ ▄ ▄▀</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">▄</span></span><span class="z-meta z-function-call z-arguments z-shell"> ▀▄▄</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">▄</span></span><span class="z-meta z-function-call z-arguments z-shell"> ▀ ▀ ▀▄▀█▄</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">▀█▄</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">▄▄▄▄▄▄▄</span></span><span class="z-meta z-function-call z-arguments z-shell"> ▄▄▄▄▄▄▄▄▄ ▄▄▄▄▄▄▄▄▄▄▄ ▄▄▄▄▄▄▄▄▄ ▀▀█</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">██████</span></span><span class="z-meta z-function-call z-arguments z-shell"> █████ ███ █████ ███ █████ ███ ▀█</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">██████</span></span><span class="z-meta z-function-call z-arguments z-shell"> █████ ███ █████ ▀▀▀ █████ ███ ▄█▄</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">██████</span></span><span class="z-meta z-function-call z-arguments z-shell"> █████ ███ █████ █████ ███ ████▄</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">██████</span></span><span class="z-meta z-function-call z-arguments z-shell"> █████ ███ █████ ▄▄▄ █████ ███ █████</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">██████</span></span><span class="z-meta z-function-call z-arguments z-shell"> █████ ███ ████ ███ █████ ███ ████▀</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">▀▀▀██▄</span></span><span class="z-meta z-function-call z-arguments z-shell"> ▀▀▀▀▀▀▀▀▀▀ ▀▀▀▀▀▀▀▀▀▀ ▀▀▀▀▀▀▀▀▀▀ ██▀</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀</span></span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">https://loco.rs</span></span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">listening</span></span><span class="z-meta z-function-call z-arguments z-shell"> on port 3000</span> </span></code></pre> <h3 id="a-taste-of-loco-example-features-and-techniques">A taste of Loco: Example Features and Techniques</h3> <p><img src="https://rng0.io/blog/introducing-loco/5.webp" alt="" /></p> <h4 id="using-the-generator-framework">Using the Generator Framework</h4> <p>You can drive all of your development by generating parts of your app. For exapmle <code>loco generate</code> will automatically create your entities and inject them into all the required places in your app.</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo loco generate model article title:string content:text</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">added:</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>migration/src/m20231202_173012_articles.rs<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">injected:</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>migration/src/lib.rs<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">injected:</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>migration/src/lib.rs<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">added:</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>tests/models/articles.rs<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">injected:</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>tests/models/mod.rs<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span></span> </span></code></pre> <p>You can use <code>generate scaffold</code> to create a complete CRUD API based on a model description that you specify on the CLI.</p> <h4 id="using-authentication">Using authentication</h4> <p>By using the <code>auth::JWT</code> <em>extractor</em> (extractors are extension points in Axum), you can opt-in and have authenticated routes:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust">async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">add</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"> <span class="z-variable z-parameter z-rust">auth</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-path z-rust">auth<span class="z-punctuation z-accessor z-rust">::</span></span>JWT, </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"> State<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-variable z-parameter z-rust">ctx</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span>: <span class="z-meta z-generic z-rust">State<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>AppContext<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"> Json<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-variable z-parameter z-rust">params</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span>: <span class="z-meta z-generic z-rust">Json<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>Params<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust">Json<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>CurrentResponse<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> we only want to make sure it exists </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">let</span> _current_user <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-keyword z-other z-rust">crate</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">models<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">users<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">Model<span class="z-punctuation z-accessor z-rust">::</span></span>find_by_pid<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-bitwise z-rust">&amp;</span>ctx<span class="z-punctuation z-accessor z-dot z-rust">.</span>db<span class="z-punctuation z-separator z-rust">,</span> <span class="z-keyword z-operator z-bitwise z-rust">&amp;</span>auth<span class="z-punctuation z-accessor z-dot z-rust">.</span>claims<span class="z-punctuation z-accessor z-dot z-rust">.</span>pid</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span>await<span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> next, update </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> homework/bonus: make a comment _actually_ belong to user (user_id) </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">let</span> <span class="z-storage z-modifier z-rust">mut</span> item <span class="z-keyword z-operator z-assignment z-rust">=</span> ActiveModel <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-operator z-range z-rust">..</span><span class="z-support z-type z-rust">Default</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>default<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> params<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">update</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-bitwise z-rust">&amp;</span><span class="z-storage z-modifier z-rust">mut</span> item</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">let</span> item <span class="z-keyword z-operator z-assignment z-rust">=</span> item<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">insert</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-bitwise z-rust">&amp;</span>ctx<span class="z-punctuation z-accessor z-dot z-rust">.</span>db</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span>await<span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-path z-rust">format<span class="z-punctuation z-accessor z-rust">::</span></span>json<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>item</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h4 id="export-user-data-with-tasks">Export user data with Tasks</h4> <p><strong>Tasks are a way to encapsulate a workflow</strong>, errand, or operators use cases that need access to a running app's infrastructure such as models and data. One example of a use case is to be able to run a report and export its data, all from the CLI.</p> <p>Here's how such a task might look like:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> find it in `src/tasks/user_report.rs` </span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust">Task <span class="z-keyword z-other z-rust">for</span></span><span class="z-meta z-impl z-rust"> <span class="z-entity z-name z-impl z-rust">UserReport</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">task</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> TaskInfo</span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> description that appears on the CLI </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> TaskInfo <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> name<span class="z-punctuation z-separator z-rust">:</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>user_report<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">to_string</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> detail<span class="z-punctuation z-separator z-rust">:</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>output a user report<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">to_string</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> variables through the CLI: </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> `$ cargo loco task name:foobar count:2` </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> will appear as {&quot;name&quot;:&quot;foobar&quot;, &quot;count&quot;:2} in `vars` </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">run</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">app_context</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>AppContext, <span class="z-variable z-parameter z-rust">vars</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-meta z-generic z-rust">BTreeMap<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span>, <span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-punctuation z-section z-group z-end z-rust">)</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">let</span> users <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">users<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">Entity<span class="z-punctuation z-accessor z-rust">::</span></span>find<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">all</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-bitwise z-rust">&amp;</span>app_context<span class="z-punctuation z-accessor z-dot z-rust">.</span>db</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span>await<span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>args: <span class="z-constant z-other z-placeholder z-rust">{vars:?}</span><span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>!!! user_report: listing users !!!<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>------------------------<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-control z-rust">for</span> user <span class="z-keyword z-operator z-rust">in</span> <span class="z-keyword z-operator z-bitwise z-rust">&amp;</span>users <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>user: <span class="z-constant z-other z-placeholder z-rust">{}</span><span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-separator z-rust">,</span> user<span class="z-punctuation z-accessor z-dot z-rust">.</span>email<span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>done: <span class="z-constant z-other z-placeholder z-rust">{}</span> users<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-separator z-rust">,</span> users<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">len</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>And now you can list it:</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo loco task</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">user_report</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-keyword z-control z-regexp z-set z-begin z-shell">[</span>output a user report<span class="z-keyword z-control z-regexp z-set z-end z-shell">]</span></span> </span></code></pre> <p>And running it is simple:</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo loco task user_report var1:val1 var2:val2 ...</span> </span></code></pre> <h4 id="automation-with-background-workers">Automation with background workers</h4> <p>You can even automate heavy tasks with a <strong>distributed background worker</strong>. Start by generating it:</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo loco generate worker report_worker</span> </span></code></pre> <p>The new worker code looks like this:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">async_trait</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-generic z-rust">Worker<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>DownloadWorkerArgs<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span> <span class="z-keyword z-other z-rust">for</span></span><span class="z-meta z-impl z-rust"> <span class="z-entity z-name z-impl z-rust">DownloadWorker</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">perform</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">args</span><span class="z-punctuation z-separator z-rust">:</span> DownloadWorkerArgs</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-punctuation z-section z-group z-end z-rust">)</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>================================================<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>Sending payment report to user <span class="z-constant z-other z-placeholder z-rust">{}</span><span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-separator z-rust">,</span> args<span class="z-punctuation z-accessor z-dot z-rust">.</span>user_guid<span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> TODO: Some actual work goes here... </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>================================================<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Now just code the logic, and run it:</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo loco start<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>server-and-worker</span></span> </span></code></pre> <p>If you want to run the worker on a separate machine or separate process:</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo loco start<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>server-and-worker</span></span> </span></code></pre> <h4 id="rendering-server-side-views">Rendering Server Side Views</h4> <p>Loco supports Tera as a template engine out of the box.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-storage z-modifier z-rust">pub</span> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">render_home</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span>ViewEngine<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-variable z-parameter z-rust">v</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span>: <span class="z-meta z-generic z-rust">ViewEngine<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>TeraView<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>impl IntoResponse<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-path z-rust">format<span class="z-punctuation z-accessor z-rust">::</span></span>render<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">view</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-bitwise z-rust">&amp;</span>v<span class="z-punctuation z-separator z-rust">,</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>home/hello.html<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> <span class="z-support z-macro z-rust">json!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h4 id="building-json-apis-with-serde">Building JSON APIs with Serde</h4> <p>You automatically generated model supports Serde out of the box. Simply return it with <code>format::json</code> to render JSON for your API.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-storage z-modifier z-rust">pub</span> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">add</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span>State<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-variable z-parameter z-rust">ctx</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span>: <span class="z-meta z-generic z-rust">State<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>AppContext<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, Json<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-variable z-parameter z-rust">params</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span>: <span class="z-meta z-generic z-rust">Json<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>Params<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust">Json<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>Model<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">let</span> <span class="z-storage z-modifier z-rust">mut</span> item <span class="z-keyword z-operator z-assignment z-rust">=</span> ActiveModel <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-operator z-range z-rust">..</span><span class="z-support z-type z-rust">Default</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>default<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> params<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">update</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-bitwise z-rust">&amp;</span><span class="z-storage z-modifier z-rust">mut</span> item</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">let</span> item <span class="z-keyword z-operator z-assignment z-rust">=</span> item<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">insert</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-bitwise z-rust">&amp;</span>ctx<span class="z-punctuation z-accessor z-dot z-rust">.</span>db</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span>await<span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-path z-rust">format<span class="z-punctuation z-accessor z-rust">::</span></span>json<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>item</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h4 id="building-chats-with-websockets">Building Chats with WebSockets</h4> <p>Loco supports <code>socketioxide</code> when you enable the <code>channels</code> feature.</p> <p>Connect your channel:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust">Hooks <span class="z-keyword z-other z-rust">for</span></span><span class="z-meta z-impl z-rust"> <span class="z-entity z-name z-impl z-rust">App</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span>... </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">routes</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">ctx</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>AppContext</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> AppRoutes</span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-path z-rust">AppRoutes<span class="z-punctuation z-accessor z-rust">::</span></span>empty<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">prefix</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>/api<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">add_app_channels</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-storage z-type z-rust">Self</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>register_channels<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>ctx</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">register_channels</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">_ctx</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>AppContext</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> AppChannels</span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">let</span> channels <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">AppChannels<span class="z-punctuation z-accessor z-rust">::</span></span>default<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> channels<span class="z-punctuation z-accessor z-dot z-rust">.</span>register<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">ns</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>/<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> <span class="z-meta z-path z-rust">channels<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">application<span class="z-punctuation z-accessor z-rust">::</span></span>on_connect</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> channels </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span></code></pre> <p>And implement your channel logic:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">on_connect</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">socket</span><span class="z-punctuation z-separator z-rust">:</span> SocketRef, Data<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-variable z-parameter z-rust">data</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span>: <span class="z-meta z-generic z-rust">Data<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>Value<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">info!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>Socket.IO connected: {:?} {:?}<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> socket<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">ns</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> socket<span class="z-punctuation z-accessor z-dot z-rust">.</span>id</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> socket<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">emit</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>auth<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> data</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> socket<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">on</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>message<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">|</span></span></span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-variable z-parameter z-rust">socket</span><span class="z-punctuation z-separator z-rust">:</span> SocketRef<span class="z-punctuation z-separator z-rust">,</span> Data::&lt;Value&gt;<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-variable z-parameter z-rust">data</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> Bin<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-variable z-parameter z-rust">bin</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-section z-parameters z-end z-rust">|</span></span> </span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">info!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>Received event: {:?} {:?}<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> data<span class="z-punctuation z-separator z-rust">,</span> bin</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> socket<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">bin</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>bin</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">emit</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>message-back<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> data</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> </span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> socket<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">on</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>message-with-ack<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">|</span></span></span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust">Data::&lt;Value&gt;<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-variable z-parameter z-rust">data</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> <span class="z-variable z-parameter z-rust">ack</span><span class="z-punctuation z-separator z-rust">:</span> AckSender<span class="z-punctuation z-separator z-rust">,</span> Bin<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-variable z-parameter z-rust">bin</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-section z-parameters z-end z-rust">|</span></span> </span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">info!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>Received event: {:?} {:?}<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> data<span class="z-punctuation z-separator z-rust">,</span> bin</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> ack<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">bin</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>bin</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">send</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>data</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> </span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></code></pre> <h4 id="seeding-data">Seeding Data</h4> <p>Loco has a mini framework for seeding databases with initial data or fixed data, use it from anywhere you like (typically from within a Loco task):</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> path <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">std<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">path<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">Path<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>src/fixtures<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-meta z-path z-rust">db<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">run_app_seed<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-generic z-rust"><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>App<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-bitwise z-rust">&amp;</span>app_context<span class="z-punctuation z-accessor z-dot z-rust">.</span>db<span class="z-punctuation z-separator z-rust">,</span> path</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span>await<span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <h4 id="piece-of-cake-deployment">Piece of Cake Deployment</h4> <p>Run <code>cargo loco deployment</code> and choose your preferred deployment stack from a list.</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo loco generate deployment</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">?</span></span><span class="z-meta z-function-call z-arguments z-shell"> ❯ Choose your deployment ›</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">❯</span></span><span class="z-meta z-function-call z-arguments z-shell"> Docker</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">❯</span></span><span class="z-meta z-function-call z-arguments z-shell"> Shuttle</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">❯</span></span><span class="z-meta z-function-call z-arguments z-shell"> Nginx</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">...</span></span> </span></code></pre> <p>For example if you choose <code>docker</code> you'll get a automagically generated <code>Dockerfile</code> specially crafted for your app. Or if you choose <code>shuttle</code>, Loco will change your app binary and configuration to fit a 1-command deploy to <a href="https://shuttle.rs">Shuttle.rs</a>.</p> <h4 id="sending-emails">Sending Emails</h4> <p>You can send emails without any external service. It includes building your own email templates and sending them out the wire.</p> <p>Like in Rails, you create mailers:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust"><span class="z-entity z-name z-impl z-rust">AuthMailer</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-documentation z-rust"><span class="z-punctuation z-definition z-comment z-rust">///</span> When email sending is failed </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-modifier z-rust">pub</span> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">send_welcome</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">ctx</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>AppContext, <span class="z-variable z-parameter z-rust">user</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-meta z-path z-rust">users<span class="z-punctuation z-accessor z-rust">::</span></span>Model</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-punctuation z-section z-group z-end z-rust">)</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">Self</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>mail_template<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> ctx<span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span>... </span></span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> </span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span>await<span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>And Loco takes care of the required background jobs to actually perform the send task.</p> <h4 id="storage-and-file-uploads">Storage and File Uploads</h4> <p>Loco contains a powerful framework for file uploads and blob storage which supports:</p> <ul> <li>Single storage</li> <li>Multiple storage providers</li> <li>Storage strategies and advanced logic for implementing primary/backup redundancy, automatic mirroring and others.</li> </ul> <p>It is available through <code>Context</code> from everywhere:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust">ctx<span class="z-punctuation z-accessor z-dot z-rust">.</span>storage </span><span class="z-source z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">as_ref</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span><span class="z-source z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">expect</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>storage<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span><span class="z-source z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">upload</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>path<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">as_path</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> <span class="z-keyword z-operator z-bitwise z-rust">&amp;</span>content</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span><span class="z-source z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span>await<span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <h2 id="conclusion-and-next-steps">Conclusion and Next Steps</h2> <h3 id="the-future-of-web-development-with-loco">The Future of Web Development with Loco</h3> <p><strong>Loco strives to be feature packed, and feature-parity with Rails</strong>, at least for the big features. Some other goals for Loco are:</p> <ul> <li><strong>Be a one person framework</strong>. For those looking to build side projects easily, quickly, and cheap, and later scale those to startups with the same stable, robust, performant codebase.</li> <li><strong>Fatigue-free maintenance</strong>. Simple dependencies, single binary, easy deployment.</li> <li>Everywhere possible - <strong>driven by tools and powertools</strong>. The Loco CLI, generators, test kits for easy testing.</li> <li><strong>Embrace modern, sparkly, Web development practices</strong>, but also <strong>old and boring</strong> practices because boring is simple. This is why Loco supports <strong>both</strong> JSON API mode with clientside frontend as well as server-side template rendering.</li> </ul> <h3 id="inviting-feedback-and-contributions">Inviting Feedback and Contributions</h3> <p><img src="https://rng0.io/blog/introducing-loco/6.webp" alt="" /></p> <p><strong>Creating a diverse committer team is very important</strong>. It allows people of all skill levels coming from any kind of programming language to share ideas and interact. We especially want to see Rubyists and Rails devs contributing ideas and code to Loco.</p> <p>Take a look at the issues, or come up with your own ideas and scratch your own itch -- <a href="https://github.com/loco-rs/loco/issues">we're accepting PRs!</a>.</p> How to easily implement a configuration-first provider pattern in Rust 🦀 Mon, 16 Oct 2023 00:00:00 +0000 Unknown https://rng0.io/blog/how-to-easily-implement-a-configurationfirst-provider-pattern-in-rust/ https://rng0.io/blog/how-to-easily-implement-a-configurationfirst-provider-pattern-in-rust/ <p><img src="https://rng0.io/blog/how-to-easily-implement-a-configurationfirst-provider-pattern-in-rust/1.png" alt="" /></p> <p>In this post we’ll review a way to <strong>build a provider pattern in Rust</strong> which is dynamic, extensible, and fun to maintain, while as statically typed as possible.</p> <p><strong>You might need a provider pattern when building a configuration for a system that has plugins</strong>, or different swappable components, logic encoded with no-code approaches, workflow configuration (think: CI/CD YAMLs), and more.</p> <p>Here’s an example YAML of two “blocks” for some kind of a workflow runner. <strong>It lets you configure which blocks run one after another and their settings</strong>.</p> <p>Below, we see an <code>env_block</code> setting environment variables and a <code>shell_block</code> running a shell command. Each block is configured via the same YAML, and, once loaded, has a single <code>run</code> function.</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">kind</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">env_block</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">id</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">env1</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">keys</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">PATH</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">foo/bar</span> </span><span class="z-source z-yaml"><span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">kind</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">shell_block</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">id</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">sh1</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">cmd</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-quoted z-double z-yaml"><span class="z-punctuation z-definition z-string z-begin z-yaml">&quot;</span>echo $PATH<span class="z-punctuation z-definition z-string z-end z-yaml">&quot;</span></span> </span></code></pre> <p><strong>We’ll implement dynamic block creation from configuration</strong>, or as some would say <em>plugin system</em>, or <em>registry of blocks</em>, or <em>strategy design pattern</em>, or <em>provider pattern</em>.</p> <p>Ultimately we want something like this:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> blocks_config <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-support z-function z-rust">load_yaml</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> box dyn for fexibility </span></span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> do something to turn config into Vec&lt;Box&lt;dyn Block&gt;&gt; </span></span><span class="z-source z-rust">blocks <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-support z-function z-rust">figure_it_out</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>blocks_config</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust">blocks<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">for_each</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">|</span></span></span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-variable z-parameter z-rust">b</span><span class="z-punctuation z-section z-parameters z-end z-rust">|</span></span> </span><span class="z-meta z-function z-closure z-rust">b<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">run</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <p>This code is completely ignorant of the different types of blocks we have, or their different settings.</p> <h2 id="the-registry">The “registry”</h2> <p><img src="https://rng0.io/blog/how-to-easily-implement-a-configurationfirst-provider-pattern-in-rust/2.png" alt="" /></p> <p><strong>Remember: we have a collection of</strong> <code>block</code> <strong>s</strong>. Each block, has a different concrete implementation, and each block can have its own configuration:</p> <ul> <li><code>env_block</code> sets a bunch of environment variables</li> <li><code>shell_block</code> runs a shell command</li> <li>any other block..</li> </ul> <p>Each block has a <code>run</code> function, available through a <code>Block</code> trait.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-storage z-type z-trait z-rust">trait</span> <span class="z-entity z-name z-trait z-rust">Block</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">run</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-punctuation z-section z-group z-end z-rust">)</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Normally we would need <strong>something</strong> that knows about all different kinds of blocks, so maybe something like a blocks registry, <code>registry.rs</code>:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> load YAML </span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">build_blocks</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">blocks_config</span><span class="z-punctuation z-separator z-rust">:</span> ..</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> </span></span><span class="z-keyword z-operator z-range z-rust">..</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-control z-rust">for</span> block <span class="z-keyword z-operator z-rust">in</span> blocks_config<span class="z-punctuation z-accessor z-dot z-rust">.</span>blocks <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-control z-rust">match</span> block<span class="z-punctuation z-accessor z-dot z-rust">.</span>kind <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-operator z-range z-rust">..</span> <span class="z-invalid z-illegal z-rust">do</span> something based on the `kind` field <span class="z-keyword z-operator z-range z-rust">..</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-operator z-range z-rust">..</span> build a block <span class="z-keyword z-operator z-range z-rust">..</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-operator z-range z-rust">..</span> push it into a vec <span class="z-keyword z-operator z-range z-rust">..</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></code></pre> <p>Some issues with this:</p> <ul> <li>Late binding: we only know what the rest of the configuration fields mean <strong>after</strong> we deal with the <code>.kind</code> attribute, so we'll have to keep it as <code>serde_json::Value</code></li> <li>Or, we’ll have to think of a different kind of configuration format which will deserialize into a strongly typed model, such as:</li> </ul> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">env_block</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-constant z-numeric z-float z-decimal z-yaml"><span class="z-punctuation z-separator z-decimal z-yaml">.</span>.</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">shell_block</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-constant z-numeric z-float z-decimal z-yaml"><span class="z-punctuation z-separator z-decimal z-yaml">.</span>.</span> </span></code></pre> <p>Keep in mind that the above solution loses ordering between <code>env_block</code> and <code>shell_block</code>. There are additional solutions, but they revolve around the same idea: encoding types in <strong>configuration structure</strong> so that deserialization has enough power to infer those types.</p> <p><strong>What ever solution we’ll find, it’s not as easy to maintain</strong> — when we need to add new kinds of blocks, when some blocks are enabled only by <em>features</em>, or when we have complex block creation logic.</p> <h2 id="serde-can-deserialize-traits-with-typetag">Serde can deserialize traits with <code>typetag</code></h2> <p><img src="https://rng0.io/blog/how-to-easily-implement-a-configurationfirst-provider-pattern-in-rust/3.png" alt="" /></p> <p>What if we can <strong>deserialize each block options</strong> into <em>some</em> <em><strong>behavior</strong></em> <em>that can build a block from the options</em>?</p> <p>The key here is <a href="https://github.com/dtolnay/typetag">typetag</a>. <code>typetag</code> is a great library that you can use to have <code>serde</code> deserialize trait objects using a single discriminating field for indicating a type, ultimately "hidden" by a trait.</p> <p><strong>And once we can deserialize trait objects, we can dial into that dynamic behavior that we wanted.</strong></p> <p>Below, our type tag in the YAML is <code>kind</code>. This is all the information that's needed to decide which implementation of <code>block</code> the next sibling fields in the YAML relate to.</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">kind</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">env_block</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">id</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">env1</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">keys</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">PATH</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">foo/bar</span> </span><span class="z-source z-yaml"><span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">kind</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">shell_block</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">id</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">sh1</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">cmd</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-quoted z-double z-yaml"><span class="z-punctuation z-definition z-string z-begin z-yaml">&quot;</span>echo $PATH<span class="z-punctuation z-definition z-string z-end z-yaml">&quot;</span></span> </span></code></pre> <p><strong>Now, we define a</strong> <code>BlockBuilder</code> <strong>trait</strong>. We say that a <code>BlockBuilder</code>'s job is to take a particular block's YAML options and configure a <code>block</code> for us based on those options, remember, it returns a <code>Box&lt;dyn Block&gt;</code> because it needs to build <em>any kind of block</em> for us, and behind the scenes it will also magically pick the concrete type for us.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">typetag</span>::<span class="z-variable z-annotation z-rust">serde</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">tag <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>kind<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-storage z-type z-trait z-rust">trait</span> <span class="z-entity z-name z-trait z-rust">BlockBuilder</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">build</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn Block<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-return-type z-rust"></span></span></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p><strong>The important bit is the</strong> <code>tag = "kind"</code> <strong>attribute</strong>, which tells <code>typetag</code> to generate the right code to build a specific <code>impl</code> of <code>BlockBuilder</code> that relates to our specific <code>block</code>.</p> <blockquote> <p><em><strong>Why a seperate abstraction for building our block?</strong></em> <em>because we recognize that a “live” block, may contain fields that are initialized but not serializable, for example a live database connection, while a block configuration section may contain a connection</em> <em><strong>string</strong></em> <em>. This way, as a best-practice, we separate the behavior of</em> <em><strong>constructing</strong></em> <em>blocks from configuration from the actual block. We’ll show an example later where we can combine the two.</em></p> </blockquote> <p>Here’s the data model for our env block options on the Rust side, as defined in the YAML file:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">derive</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">Default<span class="z-punctuation z-separator z-rust">,</span> Debug<span class="z-punctuation z-separator z-rust">,</span> Serialize<span class="z-punctuation z-separator z-rust">,</span> Deserialize<span class="z-punctuation z-separator z-rust">,</span> Clone</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-struct z-rust">struct</span> </span><span class="z-meta z-struct z-rust"><span class="z-entity z-name z-struct z-rust">EnvBlockOpts</span> </span><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">id</span><span class="z-punctuation z-separator z-type z-rust">:</span> String, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">keys</span><span class="z-punctuation z-separator z-type z-rust">:</span> <span class="z-meta z-generic z-rust">HashMap<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span>, <span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>We see that it’s an <code>env</code> block, which is one kind of block our of all possible blocks we have.</p> <p>We implement a <code>BlockBuilder</code>, and <strong>note the specific values we set for the</strong> <code>typetag::serde</code> <strong>attribute:</strong> <code>env_block</code>.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">typetag</span>::<span class="z-variable z-annotation z-rust">serde</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">name <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>env_block<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust">BlockBuilder <span class="z-keyword z-other z-rust">for</span></span><span class="z-meta z-impl z-rust"> <span class="z-entity z-name z-impl z-rust">EnvBlockOpts</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">build</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn Block<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">Box</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">EnvBlock<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-variable z-language z-rust">self</span><span class="z-punctuation z-accessor z-dot z-rust">.</span>id<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">clone</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> <span class="z-variable z-language z-rust">self</span><span class="z-punctuation z-accessor z-dot z-rust">.</span>keys<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">clone</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> The actual env block (which isn&#39;t so important for the purpose of this discussion) </span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-struct z-rust">struct</span> </span><span class="z-meta z-struct z-rust"><span class="z-entity z-name z-struct z-rust">EnvBlock</span> </span><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">id</span><span class="z-punctuation z-separator z-type z-rust">:</span> String, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">keys</span><span class="z-punctuation z-separator z-type z-rust">:</span> <span class="z-meta z-generic z-rust">HashMap<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span>, <span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust"><span class="z-entity z-name z-impl z-rust">EnvBlock</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">new</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">id</span><span class="z-punctuation z-separator z-rust">:</span> String, <span class="z-variable z-parameter z-rust">keys</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-generic z-rust">HashMap<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span>, <span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-storage z-type z-rust">Self</span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">Self</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> id<span class="z-punctuation z-separator z-rust">,</span> keys </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Now we can load all block configurators:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> `load_yaml` just reads and deserializes yaml, no special code </span></span><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> block_configs <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-support z-function z-rust">load_yaml</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> Vec&lt;Box&lt;dyn BlockBuilder&gt;&gt; </span></span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> generically call builders to get all needed blocks. </span></span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> we don&#39;t know or care about specific block implementations, </span></span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> we&#39;re getting a Box&lt;dyn Block&gt; which has a `.run` function and that&#39;s perfect for us. </span></span><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> blocks <span class="z-keyword z-operator z-assignment z-rust">=</span> block_configs<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">iter</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">for_each</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">|</span></span></span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-variable z-parameter z-rust">block_config</span><span class="z-punctuation z-section z-parameters z-end z-rust">|</span></span> </span><span class="z-meta z-function z-closure z-rust">block_config<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">build</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-meta z-path z-rust">collect<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-generic z-rust"><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-keyword z-operator z-rust">_</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <p>Note that there nothing in the code above indicating a specific kind of block or a specific implementation of a block. <strong>Everything is fully dynamic</strong> and extensible, and our code is ignorant of how many types of blocks there are, how to build them, and so on.</p> <p>To illustate what’s happening when building an <code>env_block</code>:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">YAML </span><span class="z-text z-plain">-&gt; we deserialize YAML </span><span class="z-text z-plain"> -&gt; uses `kind=env_block` to resolve EnvBlockOpts </span><span class="z-text z-plain"> -&gt; deserialized EnvBlockOpts into a &lt;dyn BlockBuilder&gt; trait which exposes a `build` function </span><span class="z-text z-plain"> -&gt; we call `.build` </span><span class="z-text z-plain"> -&gt; get a fully configured &lt;dyn Block&gt; </span></code></pre> <h2 id="nested-blocks-and-nested-configuration">Nested blocks and nested configuration</h2> <p><strong>What if we have a block that contains a block, and nests its YAML configuration as well?</strong></p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-comment z-line z-number-sign z-yaml"><span class="z-punctuation z-definition z-comment z-line z-number-sign z-yaml">#</span> a top level ping block </span></span><span class="z-source z-yaml"><span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">kind</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">ping_block</span> </span><span class="z-source z-yaml"> <span class="z-comment z-line z-number-sign z-yaml"><span class="z-punctuation z-definition z-comment z-line z-number-sign z-yaml">#</span> billing block </span></span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">billing</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">addr</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-constant z-numeric z-float z-decimal z-yaml"><span class="z-punctuation z-separator z-decimal z-yaml">.</span>.</span> </span><span class="z-source z-yaml"> <span class="z-comment z-line z-number-sign z-yaml"><span class="z-punctuation z-definition z-comment z-line z-number-sign z-yaml">#</span> auth block </span></span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">auth</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">addr</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-constant z-numeric z-float z-decimal z-yaml"><span class="z-punctuation z-separator z-decimal z-yaml">.</span>.</span> </span></code></pre> <p>Where a <code>ping_block</code>'s <code>.run</code> calls both children blocks's <code>.run</code> in order to accomplish what it needs to do.</p> <p>The full implementation of dynamically loading <code>ping_block</code> would be:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">typetag</span>::<span class="z-variable z-annotation z-rust">serde</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">tag <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>kind<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-storage z-type z-trait z-rust">trait</span> <span class="z-entity z-name z-trait z-rust">BlockBuilder</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">build</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn Block<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-return-type z-rust"></span></span></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> note: billing and auth config builders will be automatically resolved by typetag but </span></span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> `.build` will not be automatically called for the nested blocks </span></span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">derive</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">Serialize<span class="z-punctuation z-separator z-rust">,</span> Deserialize</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-storage z-type z-struct z-rust">struct</span> </span><span class="z-meta z-struct z-rust"><span class="z-entity z-name z-struct z-rust">PingBlockOpts</span> </span><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">billing</span><span class="z-punctuation z-separator z-type z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn BlockBuilder<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">auth</span><span class="z-punctuation z-separator z-type z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn BlockBuilder<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">derive</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">Serialize<span class="z-punctuation z-separator z-rust">,</span> Deserialize</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-storage z-type z-struct z-rust">struct</span> </span><span class="z-meta z-struct z-rust"><span class="z-entity z-name z-struct z-rust">BillingOpts</span> </span><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">addr</span><span class="z-punctuation z-separator z-type z-rust">:</span> String </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">derive</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">Serialize<span class="z-punctuation z-separator z-rust">,</span> Deserialize</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-storage z-type z-struct z-rust">struct</span> </span><span class="z-meta z-struct z-rust"><span class="z-entity z-name z-struct z-rust">AuthOpts</span> </span><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">auth</span><span class="z-punctuation z-separator z-type z-rust">:</span> String </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> this is our actual ping block, which is composed out of two other blocks </span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-storage z-type z-struct z-rust">struct</span> </span><span class="z-meta z-struct z-rust"><span class="z-entity z-name z-struct z-rust">PingBlock</span> </span><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">billing_ping</span><span class="z-punctuation z-separator z-type z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn Block<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">auth_ping</span><span class="z-punctuation z-separator z-type z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn Block<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-storage z-type z-struct z-rust">struct</span> </span><span class="z-meta z-struct z-rust"><span class="z-entity z-name z-struct z-rust">BillingBlock</span> </span><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> .. </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-storage z-type z-struct z-rust">struct</span> </span><span class="z-meta z-struct z-rust"><span class="z-entity z-name z-struct z-rust">AuthBlock</span> </span><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> .. </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"> </span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> individual block builders, with their appropriate opts: </span></span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">typetag</span>::<span class="z-variable z-annotation z-rust">serde</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">name <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>auth_block<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust">BlockBuilder <span class="z-keyword z-other z-rust">for</span></span><span class="z-meta z-impl z-rust"> <span class="z-entity z-name z-impl z-rust">AuthOpts</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">build</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn Block<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-operator z-range z-rust">..</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">typetag</span>::<span class="z-variable z-annotation z-rust">serde</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">name <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>billing_block<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust">BlockBuilder <span class="z-keyword z-other z-rust">for</span></span><span class="z-meta z-impl z-rust"> <span class="z-entity z-name z-impl z-rust">BillingOpts</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">build</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn Block<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-operator z-range z-rust">..</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> The key is implementing a nested `build` in the top level block builder, </span></span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> which calls individual nested block builders manually </span></span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">typetag</span>::<span class="z-variable z-annotation z-rust">serde</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">name <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>ping_block<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust">BlockBuilder <span class="z-keyword z-other z-rust">for</span></span><span class="z-meta z-impl z-rust"> <span class="z-entity z-name z-impl z-rust">PingBlockOpts</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">build</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn Block<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> let its inner blocks config builders build the actual blocks </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> and save them as fields via constructor </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">Box</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">PingBlock<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-variable z-language z-rust">self</span><span class="z-punctuation z-accessor z-dot z-rust">.</span>billing<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">build</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> <span class="z-variable z-language z-rust">self</span><span class="z-punctuation z-accessor z-dot z-rust">.</span>auth<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">build</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h2 id="self-building-blocks">Self-building blocks</h2> <p><strong>Sometimes a block can have serializable fields</strong> and its construction is very simple.</p> <p>In that case we can save a bit of typing and have a bloc implement <strong>both</strong> <code>BlockBuilder</code> and the <code>Block</code> trait so it can build itself with no need for a seperate "opts" struct.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> the block struct itself is serializable </span></span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">derive</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">Serialize<span class="z-punctuation z-separator z-rust">,</span> Deserialize</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-storage z-type z-struct z-rust">struct</span> </span><span class="z-meta z-struct z-rust"><span class="z-entity z-name z-struct z-rust">HealthBlock</span> </span><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">addr</span><span class="z-punctuation z-separator z-type z-rust">:</span> String </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> note we implement a config builder for HealthBlock itself, there&#39;s no separate configuration opts struct </span></span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">typetag</span>::<span class="z-variable z-annotation z-rust">serde</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">name <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>health_block<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust">BlockBuilder <span class="z-keyword z-other z-rust">for</span></span><span class="z-meta z-impl z-rust"> <span class="z-entity z-name z-impl z-rust">HealthBlock</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">build</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn Block<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">Box</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">HealthBlock<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-range z-rust">..</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust">Block <span class="z-keyword z-other z-rust">for</span></span><span class="z-meta z-impl z-rust"> <span class="z-entity z-name z-impl z-rust">HealthBlock</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-operator z-range z-rust">..</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> and now: </span></span><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> builder <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-support z-function z-rust">load_yaml</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-range z-rust">..</span>block yaml<span class="z-keyword z-operator z-range z-rust">..</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> this is the config builder trait </span></span><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> block <span class="z-keyword z-operator z-assignment z-rust">=</span> builder<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">build</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> which builds itself </span></span></code></pre> <p>This is a compact way to implement this pattern, and you can start with it by default.</p> Errors in Rust: A Formula Fri, 13 Oct 2023 00:00:00 +0000 Unknown https://rng0.io/blog/errors-in-rust-a-formula/ https://rng0.io/blog/errors-in-rust-a-formula/ <p><img src="https://rng0.io/blog/errors-in-rust-a-formula/1.png" alt="" /></p> <p><strong>Errors are a multi layer cake</strong>. Errors in Rust have a <em>taste</em> factor because it falls in the “there is more than one way to do it” language camp. It mixes-in ML (as in: Ocaml), functional, and algol-like flavors for its constructs, and</p> <p>takes the best ideas from other languages, and it has strict opinions on everything</p> <p>safety: concurrency, memory access, sharing resources, and more.</p> <p>This is why <strong>error handling in Rust is a multi layer cake</strong>.</p> <blockquote> <p>→ → →</p> <p>🙋‍♀️ Have questions? Want to share your experience?, <a href="https://twitter.com/jondot">follow me on Twitter</a> 😎</p> </blockquote> <p><em>PS: if you’re impatient you can skip to the bottom to review the formula/checklist for errors in Rust.</em></p> <h2 id="choosing-the-right-error-handling-philosophy">Choosing the Right Error Handling Philosophy</h2> <p><strong>As a contrast, look at Go</strong>, which takes an arguably correct approach, it says:</p> <p><em>“An error is a simple thing. Handle it at the callsite or bust”</em>. And look at Java, which says <em>“Throw it and someone</em></p> <p><em>else will handle it”</em> which, as history proves, can be very wrong, many times.</p> <p>The trouble is that <strong>when errors are <em>precieved</em> to be a simple thing like in Go</strong>, they’re often a string, or an object that by the time it reaches the handler, it lacks context or information to make a smart error recovery decision. And with Java’s approach, an error is thrown and the stack is busted, add in a good measure of lack of best practices, and for each layer of bubbling up, you lose more and more context, and again — <strong>by the time the handler gets an error, they are in a bad position to make an informed decision</strong>.</p> <h2 id="an-error-philosophy-in-rust">An Error Philosophy in Rust</h2> <p>The basics of errors in Rust are these:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> define a Result type as a convenience, already encoding-in our Error type </span></span><span class="z-source z-rust"><span class="z-storage z-type z-type z-rust">type</span> <span class="z-entity z-name z-type z-rust">Result</span><span class="z-keyword z-operator z-comparison z-rust">&lt;</span>T<span class="z-keyword z-operator z-comparison z-rust">&gt;</span> <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>T, ParserError<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"> </span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> an Error is a type, nothing special </span></span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">derive</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">Debug</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-storage z-type z-enum z-rust">enum</span> <span class="z-entity z-name z-enum z-rust">ParserError</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> EmptyText<span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> Parse<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">String</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> return a result, which may or may not be an error. to find it `match` on its content </span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">parse</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span>..</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>AST<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-operator z-range z-rust">..</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>You have a <code>Result</code> type, which encodes the <code>Ok</code> and <code>Err</code> states, which are represented as types you can create. The error type can be anything, but <strong>an</strong> <code>enum</code> <strong>is a smart idea (and the most popular one)</strong>.</p> <p>A function returns a <code>Result</code>.</p> <p><strong>For managing errors</strong>, Rust picks up on the gaps of other languages, and tells you:</p> <ul> <li>Handle errors on callsite, or</li> <li>Bubble up errors <em>in an orderly fashion</em> (also means: no throw/catch mechanism like in Java)</li> </ul> <p><strong>For handling errors</strong> Rust also gives you everything you need, as part of the language. <strong>Handling</strong> errors is first-class:</p> <ul> <li>Errors are <strong>encouraged</strong> to be richly typed (e.g. unlike Go)</li> <li>You are encouraged to <code>match</code> <strong>on errors</strong>, and Rust acknowledges that errors and failures are a <strong>rich experience, that can easily become a frustrating one</strong> and gives you ergonomics for those not so trivial error handling situations. Again, unlike Go.</li> <li><strong>Passing errors around</strong> is similar to returning results from functions, which is what a programming language already knows how to do. Like Go, but unlike Java. It means that the programmer should not be caught off guard, and that every single powerful construct a programmer has to create or handle function results (aggregation, expression building, conversion, concurrency, etc.) is available to handle errors, unlike Go and unlike Java.</li> </ul> <p>Because an error is just a return value, you can model this in any other programming language, right?</p> <p><em>I can model errors as result types in Java, whats the big deal?</em> and the big deal is how everything connects — when the standard library is all oriented towards this, the syntax factors-in errors as results, and the community and mindset is already there as well your get error-story-nirvana.</p> <h2 id="rant-errors-are-not-simple-rant"><code>&lt;rant&gt;</code> Errors are Not Simple <code>&lt;/rant&gt;</code></h2> <p><strong>While this is not to dis Go</strong>, the difference between Go and Rust is a great illustration of how “language thinking” (e.g. Wharf theorem) affects code and cognitive load.</p> <p>Every person that migrated from Go to Rust that I’ve met — told me</p> <p>they can’t go back. That they can’t believe how they miscalculated how complex errors were, and that today they can see the amount of edge cases that their Go code was ignorant of.</p> <p><strong>It’s all about the <em>cognitive load weight transfer</em></strong>: in C, errors are simple because C was an assembly preprocessor. That’s why the complexity is shifted towards the developer. In Go, which aims to classically improve on this, but still be inspired from the simplicity, the weight is shifted towards the developer as well. The difference is that in C you knew you have no safety net.</p> <p>This is why they couldn’t believe how they were “sold” on the idea that errors are “one dimensional”, simple, (e.g. handle it or just bork, using the typical <code>if err != nil</code> muscle reflex you get in Go). Failures are not simple. <strong>Error cases are not trivial</strong>. This is why we have spaceships crash, people.</p> <h2 id="there-s-no-free-lunch">There’s no Free Lunch</h2> <p><strong>Errors in Rust are a learning experience</strong> and a powerful tool. Errors aren’t that pesky thing you need to deal with, and a master of errors is a master of the language.</p> <p>Rust offers tools to build a great error modeling and handling story that can power mission-critical software — if that’s your thing. For me, eventhough I don’t build software for spaceships and neuclear reactors, I always enjoyed building reliable software that doesn’t wake you up at night.</p> <p>This is why errors in Rust require more investment from your side.</p> <ul> <li><strong>Creating errors has more cognitive load</strong>. You can’t just <code>new Error("oops")</code> or <code>return "can't load"</code> or <code>throw new Foobar()</code>. Errors need to be created as a type and conform to a few rules.</li> <li><strong>Connect with a failure story</strong>: to provide backtrace, root cause</li> <li><strong>Connect with a user experience</strong>: provide <em>debug</em> and <em>display</em> implementation which cater to <em>operator</em> and <em>end user</em> respectively.</li> <li><strong>Convert results sensibly and responsibly</strong>: you’ll quickly realize that a function returning <code>Result&lt;(), ErrA&gt;</code> does not work with an inner function returning <code>Result&lt;(), ErrB&gt;</code>.</li> <li><strong>Handle the complexity of a real system</strong>, where you can get a multi-layered error: <code>Error(DatabaseError(ConnectionError(PoolError(reason))))</code>, <strong>each layer of that error onion, has a fork and a decision you need to explicitly code</strong>. This is a Good Thing, unless the library author did a lousy job modeling errors, in which case you're just compensating for their gaps, which is also a bug-preventing action, in anycase.</li> <li><strong>Be responsible towards others</strong>. The errors you return may be consumed by others. You need to think with intent about what kind of error story <em>they</em> will have.</li> </ul> <h2 id="structure-semantics-and-user-experience">Structure, semantics, and user experience</h2> <p>We’re going to review some practical patterns, and design challenges. Everything here will be built <strong>with just two error libraries</strong>.</p> <ol> <li><code>thiserror</code> - for libraries and core</li> <li><code>eyre</code> (which is similar / drop-in alternative to <code>anyhow</code>) - optional, if you're targeting apps (CLI mostly)</li> </ol> <p>We’ll review:</p> <ul> <li><strong>How to create and add context</strong> to our own errors</li> <li>How to <strong>layer, nest, and wrap</strong> errors</li> <li><strong>Map errors</strong> from dependencies into our own errors in a <code>1:1</code>, <code>1:N</code>, <code>N:1</code> relationship</li> </ul> <h2 id="creating-errors">Creating errors</h2> <p>By keeping an enum form, you can create <strong>an error hierarchy that keeps context</strong>:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-storage z-type z-enum z-rust">enum</span> <span class="z-entity z-name z-enum z-rust">ParserError</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> EmptyFile<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">String</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> the file path </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> Json<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">serde_json<span class="z-punctuation z-accessor z-rust">::</span></span>Error</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> the original JSON parser error </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> SyntaxError <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> line<span class="z-punctuation z-separator z-rust">:</span> <span class="z-support z-type z-rust">String</span><span class="z-punctuation z-separator z-rust">,</span> pos<span class="z-punctuation z-separator z-rust">:</span> <span class="z-storage z-type z-rust">usize</span> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> a complex error structure </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>When we want to create errors, or return errors we want to keep the following principles:</p> <ul> <li><strong>Type less</strong>: use <code>From</code> extensively and automatic conversions using <code>?</code></li> <li><strong>Use a ‘for-in’ loop</strong> instead of mapping for short-circuiting</li> <li><strong>Collect</strong> <code>Result&lt;Vec&gt;</code> for aggregation, instead of Vec&lt;Result&lt;..&gt;&gt;</li> </ul> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> res<span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-keyword z-operator z-rust">_</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span> <span class="z-keyword z-operator z-assignment z-rust">=</span> foo<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">map</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-range z-rust">..</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">collect</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <ul> <li><strong>Keep the source errors</strong> at all times, when possible</li> <li>In <code>Error</code>, <strong>both display and debug are important</strong>: audience; display is for end users (e.g. redact secrets, make a long text shorter), and debug is for operators (e.g. troubleshooting and logging and diagnostics).</li> <li><strong>No</strong> <code>expect</code> <strong>, no</strong> <code>unwrap</code> <strong>, no</strong> <code>panic</code> in your code unless it's a must (and mostly, due to a gap in one of the external libraries)</li> </ul> <h2 id="crate-level-and-module-level-errors">Crate-level and Module-level Errors</h2> <p>While you can definitely have a single <code>Result</code> type, with a single <code>Error</code> type for your entire crate, <strong>often you can benefit from sub-dividing your Error type</strong> into sub-errors relevant to your individual crate modules. When using a monorepo with multiple-crate workspace, this is inevitable.</p> <p>The meaning of this is that you’ll also be sub-dividing your single <code>Result</code> type to many different <code>Result</code> types relevant to your sub-modules.</p> <p>For example:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">compiler/ -&gt; (1) Result&lt;T,E&gt; = Result&lt;.., Error&gt; </span><span class="z-text z-plain"> parser/ -&gt; (2) Result&lt;T,E&gt; = Result&lt;.., ParserError&gt; </span><span class="z-text z-plain"> scanner/ -&gt; (3) Result&lt;T,E&gt; = Result&lt;.., ScannerError&gt; </span></code></pre> <p><strong>Crate-level error type</strong> is (1)</p> <p><strong>Module-level error type</strong> is (2) and (3)</p> <p>The three different <code>Result</code> types <strong>are <em>different types</em></strong>. So when returning a <code>Result&lt;.., ParserError&gt;</code> to a <code>compiler</code> function expecting a <code>Result&lt;.., Error&gt;</code> your code will break.</p> <p>And so, you’ll need to <code>.map_err</code> if the Result <code>T</code> is the same, or re-wrap your <code>Result</code> while passing the value upwards from downstream ( <code>parser</code>) to upstream ( <code>compiler</code>) dependencies.</p> <h2 id="mapping-errors">Mapping Errors</h2> <p><strong>Sometimes handling errors is simply mapping them</strong> into a different error kind, which means picking the original error apart, taking some context from it, or wrapping it as-is into a different error kind.</p> <p><strong>This requires a thoughful base error to start with</strong>. I find that most of the time it’s great to have this as a starting point (and I tend to copy it for every new module I create).</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">derive</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">thiserror::Error<span class="z-punctuation z-separator z-rust">,</span> Debug</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-enum z-rust">enum</span> <span class="z-entity z-name z-enum z-rust">Error</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">error</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>{0}<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> Message<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">String</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> a central IO wrapper </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">error</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">transparent</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-constant z-other z-rust">IO</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">from</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> <span class="z-meta z-path z-rust">std<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">io<span class="z-punctuation z-accessor z-rust">::</span></span>Error</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> will be used with `.map_err(Box::from)?;` </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">error</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">transparent</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> Any<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">from</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn <span class="z-meta z-path z-rust">std<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">error<span class="z-punctuation z-accessor z-rust">::</span></span>Error <span class="z-keyword z-operator z-rust">+</span> <span class="z-support z-type z-rust">Send</span> <span class="z-keyword z-operator z-rust">+</span> <span class="z-support z-type z-rust">Sync</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> some other common conversions </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">error</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">transparent</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> Pattern<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">from</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> <span class="z-meta z-path z-rust">regex<span class="z-punctuation z-accessor z-rust">::</span></span>Error</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">error</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">transparent</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> Json<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">from</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> <span class="z-meta z-path z-rust">serde_json<span class="z-punctuation z-accessor z-rust">::</span></span>Error</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Wrapping <code>serde_json::Error</code> through <code>Error::Json</code> feels like repeating existing information and not adding any value. Are we just repeating stuff?</p> <p>Well, there are two things happening here:</p> <ol> <li><strong>Consolidating different error types</strong>, possibly from different libraries, as one unified enum error type, which streamlines <code>Result</code> types significantly - this is <em>key</em> for useful error handling of your library by third parties. When you use a library, it's best to expect <strong>just one error type</strong> which tells you about <strong>all of the errors</strong> that are possible in this library.</li> <li><strong>Enjoying automatic conversions</strong> with <code>#[from]</code> to clean up your code, save typing, and save maintenance, by side-stepping the error conversion decision point. You can always go "manual" and remove <code>#[from]</code>, and make the conversion at each code point in your code base yourself.</li> <li><strong>Keeping an escape hatch</strong> for those <em>anyhow</em> moments. Don’t care about an error? don’t know what to do with it? the library authors made it impossible to work with? Do this karate-chop: <code>.map_err(Box::from)?;</code> and wrap it with your own accessible <code>Error</code> type.</li> </ol> <h2 id="mapping-n-1">Mapping: N:1</h2> <p>Mapping many 3rd party error types to one of your error types: when would you want to do this?</p> <ul> <li><strong>When one or more libraries have a too fine level</strong> of details in their errors. For example, when <code>Error::HttpThrottling</code>, <code>Error::RateLimit</code>, and <code>Error::AccountDepleated</code> are all the same, indicating: <em>"you're out of your API credits, or you're abusing your credits -- chill out!"</em>.</li> <li><strong>When you already know it’s game over</strong>. Knowing the specifics of the error won’t help your service. For example <code>Error::DiskFull</code>, <code>Error::CorruptPartition</code>, etc. Just wrap them under an <code>MyError::Fatal</code> and keep the orignal error in it, boxed, for more detail.</li> <li><strong>When multiple different libraries are doing the same thing</strong>, and implement a provider architecture where you can swap different providers implementing a <code>trait</code>. For example, an <code>Error::PostgresConnection</code>, <code>Error::MySqlConnectionPool</code>, with a swappable DB provider in <strong>your</strong> crate, might mean just a <code>MyError::Connection</code> for you. Remember <strong>if the trait needs to be universal over providers, errors returned in trait functions should be too</strong></li> </ul> <p>You have two ways to create this kind of mapping.</p> <h2 id="1-n-1-mapping-layer-things-and-bubble-up">1. N:1 Mapping — layer things and bubble up</h2> <p>Essentially we want to create <strong>a first-level aggregate error type, and a top-level aggregate of errors</strong>.</p> <p>Say you have database providers, and then you have your crate which does data access.</p> <p>First, create a first-level error type:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-storage z-type z-enum z-rust">enum</span> <span class="z-entity z-name z-enum z-rust">DbProviderError</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> all of these are invariably common to all database </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> providers you&#39;re dealing with </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> Connection<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-range z-rust">..</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> PoolLimit<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-range z-rust">..</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> SqlSyntax<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-range z-rust">..</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Having a trait for these providers, will return the above error, to align <strong>all the specifics</strong> of each and every different error from the different providers:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-storage z-type z-trait z-rust">trait</span> <span class="z-entity z-name z-trait z-rust">DbProvider</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">connect</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span>..</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-punctuation z-section z-group z-end z-rust">)</span>, DbProviderError<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-return-type z-rust"></span></span></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Finally, your crate which uses <code>DbProvider</code>, sets up the right provider, etc., needs to be able to accept <code>DbProviderError</code>:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-storage z-type z-enum z-rust">enum</span> <span class="z-entity z-name z-enum z-rust">MyError</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span>.. </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> Message<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">String</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">error</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">transparent</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-constant z-other z-rust">DB</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">from</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> DbProviderError</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Now, using <code>?</code> to convert a DB error to a crate-level error should create a nice and tidy error story.</p> <p><strong>NOTE</strong>: <strong>there will be cases where the two result types of your trait and your crate-level result type will not be compatible</strong> and converting through <code>?</code> will not work. That's where you'll have to manually call <code>.into()</code> on a <code>DbProviderError</code> to turn it into a <code>MyError</code> when doing a <code>.map_err</code> or creating a new <code>DbProviderError</code> yourself.</p> <p><strong>An example</strong>:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">embed_inputs</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">inputs</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>[<span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span>]</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-storage z-type z-rust">f32</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Err</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">EmbeddingError<span class="z-punctuation z-accessor z-rust">::</span></span>Other<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-group z-rust"> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>an embedding cannot be made<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">to_string</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-group z-rust"> </span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">into</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Here, <code>embed_inputs</code> is in an <code>embedding</code> module, and it has its own error hierarchy and story, and its own <code>Error</code> type.</p> <p>But, <strong>it returns a higher crate-level</strong> <code>Result</code>, and while it contains a crate-level <code>Error</code> which can convert an <code>EmbeddingError</code> with a <code>from</code> trait, it cannot be inferred automatically, so we're using <code>into()</code> on our <code>Err</code> directive.</p> <p>Another way to manually convert is to call up the <code>into</code> trait directly, and then use a <code>?</code> conversion:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">embed_inputs</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">inputs</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>[<span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span>]</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-storage z-type z-rust">f32</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">let</span> res <span class="z-keyword z-operator z-assignment z-rust">=</span> provider </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">do_something</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">map_err</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">Into</span><span class="z-punctuation z-separator z-rust">:</span><span class="z-punctuation z-separator z-rust">:</span><span class="z-meta z-generic z-rust"><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-invalid z-illegal z-rust">.</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span>error <span class="z-storage z-type z-type z-rust">type</span><span class="z-keyword z-operator z-range z-rust">..</span><span class="z-keyword z-operator z-comparison z-rust">&gt;</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>into</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>res<span class="z-keyword z-operator z-rust">?</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h3 id="a-note-about-folder-and-module-structure">A note about folder and module structure</h3> <p><strong>When dealing with providers and providers implementing traits</strong>, we many times have to map N error types and variants into 1 variant of our own.</p> <p>In that case we can create a similar looking error architecture, where each layer knows its own concrete errors.</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">root/ </span><span class="z-text z-plain"> error.rs </span><span class="z-text z-plain"> providers/ </span><span class="z-text z-plain"> error.rs </span><span class="z-text z-plain"> providerA/ </span><span class="z-text z-plain"> error.rs </span><span class="z-text z-plain"> providerB/ </span><span class="z-text z-plain"> error.rs </span><span class="z-text z-plain"> ... </span></code></pre> <p>And then next step is to offer <strong>encapsulated error types and module-local conversions</strong>:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">root/ </span><span class="z-text z-plain"> error.rs </span><span class="z-text z-plain"> providers/ </span><span class="z-text z-plain"> error.rs </span><span class="z-text z-plain"> .. { </span><span class="z-text z-plain"> ProviderA(provider_a::error::ProviderAError) </span><span class="z-text z-plain"> ProviderB </span><span class="z-text z-plain"> } </span><span class="z-text z-plain"> providerA/ </span><span class="z-text z-plain"> error.rs </span><span class="z-text z-plain"> .. { </span><span class="z-text z-plain"> SqlConnectionError(extlib::conn:Error) </span><span class="z-text z-plain"> DataTransferError(extlib::conn:Error) </span><span class="z-text z-plain"> } </span><span class="z-text z-plain"> providerB/ </span><span class="z-text z-plain"> error.rs </span><span class="z-text z-plain"> ... </span></code></pre> <p>But often, <strong>we want to “group” provider errors without caring for the details</strong> inside each of those errors, because it’s too low level and because a user cannot handle those, we just have a provider error. In essence we’re grouping errors <code>N to 1</code>.</p> <h2 id="2-n-1-mapping-put-it-in-a-box">2. N:1 Mapping — put it in a box</h2> <p><strong>If you don’t care about the specific DB provider error type</strong>, you can wrap and box it, and send it up with a <code>MyError</code> crate-level error:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-storage z-type z-enum z-rust">enum</span> <span class="z-entity z-name z-enum z-rust">MyError</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span>.. </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> Message<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">String</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-constant z-other z-rust">DB</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn <span class="z-meta z-path z-rust">std<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">error<span class="z-punctuation z-accessor z-rust">::</span></span>Error <span class="z-keyword z-operator z-rust">+</span> <span class="z-support z-type z-rust">Send</span> <span class="z-keyword z-operator z-rust">+</span> <span class="z-support z-type z-rust">Sync</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>And then, <code>.map_err(Box::from).map_err(|e| MyError::DB(e))</code>. Note that we're being very explicit here with <code>.map_err</code>, to provide space for more variants that are <code>Box&lt;dyn Error&gt;</code> under the <code>MyError</code> type.</p> <h2 id="mapping-1-1">Mapping 1:1</h2> <p>Mapping a single 3rd party error to a one of our own error variants (for the sake of this discussion we treat things like <em>stdlib</em> as 3rd party as well).</p> <p>This lets us map into <strong>our error variants</strong> which allow us to jump through error hierarchies as they exist in our modules and crates.</p> <p>For example for bubbling up through layers, viewed as a set of abstract actions:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">3rd-party error -&gt; (wrap!) -&gt; ModuleError -&gt; (wrap!) -&gt; CrateError </span></code></pre> <p>Taking a concrete parser example and viewed as a tree:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> crate </span></span><span class="z-source z-rust"><span class="z-meta z-path z-rust">ParserError<span class="z-punctuation z-accessor z-rust">::</span></span>Invalid<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> module </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-meta z-path z-rust">ScannerError<span class="z-punctuation z-accessor z-rust">::</span></span>BadInput<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-group z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> 3rd party </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-group z-rust"> <span class="z-meta z-path z-rust">Regex<span class="z-punctuation z-accessor z-rust">::</span></span>Error<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-range z-rust">..</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-group z-rust"> </span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></code></pre> <p>Jumping through layers in code:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">parse</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span>..</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span>, ParserError<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-function z-rust">scan</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> module error -&gt; crate error </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"> </span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">scan</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span>..</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span>, ScannerError<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-function z-rust">scan_with_regex</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> lib error -&gt; module error </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"> </span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">scan_with_regex</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span>..</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span>, <span class="z-meta z-path z-rust"><span class="z-storage z-type z-rust"><span class="z-storage z-type z-rust">Regex</span><span class="z-punctuation z-accessor z-rust">::</span></span></span>Error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-operator z-range z-rust">...</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Do you <strong>need</strong> all these layer? many times you don’t. But, understanding this basic structuring of errors will let you understand other libraries, and you’ll be able to “cut out” the stuff you need from this bigger picture when you need it.</p> <h2 id="mapping-1-n">Mapping 1:N</h2> <p>This happens when you get one type of error and you need more granularity in your own code. A common example is an <code>HTTPError</code> that treats everything like an error <strong>but you know</strong> that a <code>404</code> is different than a <code>500</code>, so you want different error handling strategies.</p> <p>Rust does a great job by giving you a <code>.map_err</code> precisely for this. And with a <code>match</code> clause it's also ergonomically enjoyable:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">map_err</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">|</span></span></span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-variable z-parameter z-rust">e</span><span class="z-punctuation z-section z-parameters z-end z-rust">|</span></span> </span><span class="z-meta z-function z-closure z-rust"><span class="z-keyword z-control z-rust">match</span> e <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> use e.code to create your error variants </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></code></pre> <h2 id="error-tricks">Error tricks</h2> <h3 id="ad-hoc-into">Ad-hoc <code>into</code></h3> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">do_something</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span>..</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-function z-rust">foobar</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-range z-rust">..</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">map_err</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Into</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>ModuleError<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>into</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-operator z-range z-rust">...</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h3 id="2-level-from">2-level <code>From</code></h3> <p>Some times jumping up two layers of errors can be done at every callsite, but when it’s done enough times, it’s better to refactor it out into a <code>From</code> trait. This cleans up your code and centralizes your error decision making efficiently.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">From</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>lib error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span> <span class="z-keyword z-other z-rust">for</span></span><span class="z-meta z-impl z-rust"> <span class="z-entity z-name z-impl z-rust">Error</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">from</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">e</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>lib error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-storage z-type z-rust">Self</span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">Self</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>SomeCrateError<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust"><span class="z-keyword z-other z-rust">super</span><span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">ModuleError<span class="z-punctuation z-accessor z-rust">::</span></span>SomeModuleError<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-range z-rust">...</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"> </span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> some other place </span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">do_something</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span>..</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-function z-rust">foobar</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-range z-rust">..</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-operator z-range z-rust">...</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h3 id="quick-box">Quick box</h3> <p>If you have this kind of error:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">derive</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">thiserror::Error<span class="z-punctuation z-separator z-rust">,</span> Debug</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-storage z-type z-enum z-rust">enum</span> <span class="z-entity z-name z-enum z-rust">MyError</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">error</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>{0}<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> Message<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">String</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> note this variant </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">error</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">transparent</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> Any<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">from</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn <span class="z-meta z-path z-rust">std<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">error<span class="z-punctuation z-accessor z-rust">::</span></span>Error <span class="z-keyword z-operator z-rust">+</span> <span class="z-support z-type z-rust">Send</span> <span class="z-keyword z-operator z-rust">+</span> <span class="z-support z-type z-rust">Sync</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Then you can do a quick karate-chop to convert what ever error to a <code>MyError::Any</code>.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-support z-function z-rust">foo</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-range z-rust">..</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">map_err</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">Box</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>from</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <p>If you use <code>anyhow</code>, using <code>MyError::Any</code> is a good alternative to avoid adding up another library.</p> <h2 id="handling-errors">Handling errors</h2> <p><img src="https://rng0.io/blog/errors-in-rust-a-formula/2.png" alt="" /></p> <h2 id="match-map-wrap">Match, map, wrap</h2> <p>Handling errors from the AWS SDK crate, we want to say that a missing parameter in <code>ssm</code> is OK and not an error case when deleting a parameter:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">handle_delete</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">e</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-generic z-rust">SdkError<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>DeleteParameterError<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, <span class="z-variable z-parameter z-rust">pm</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>PathMap</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-punctuation z-section z-group z-end z-rust">)</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-control z-rust">match</span> e<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">into_service_error</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-path z-rust">DeleteParameterError<span class="z-punctuation z-accessor z-rust">::</span></span>ParameterNotFound<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">_</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> <span class="z-keyword z-operator z-rust">=&gt;</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> we&#39;re ok </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> e <span class="z-keyword z-operator z-rust">=&gt;</span> <span class="z-support z-type z-rust">Err</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-other z-rust">crate</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">Error<span class="z-punctuation z-accessor z-rust">::</span></span>DeleteError <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> path<span class="z-punctuation z-separator z-rust">:</span> pm<span class="z-punctuation z-accessor z-dot z-rust">.</span>path<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">to_string</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> msg<span class="z-punctuation z-separator z-rust">:</span> e<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">to_string</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <ul> <li> <p><strong>Match</strong> — pick out the various error cases from the main <code>Error</code> type</p> </li> <li> <p><strong>Map</strong> — create a different semantic for the original Result type (mapping an error case to an <code>Ok</code> case)</p> </li> <li> <p><strong>Wrap</strong> — return a streamlined, familiar <code>Error</code> type that our crate will expose to end users</p> <p>composability</p> </li> </ul> <p>As a general rule of thumb, handling errors will always be a selection of (1) match, (2) map, (3) wrap, or all of those combined.</p> <h2 id="exit-handling">Exit handling</h2> <p>In many cases, you have no way to actually recover from an error. So the best you can do is print it out, and signal it appropriately (e.g. using proper Unix exit codes).</p> <p>Here’s one example of such handling, where a program can report an <strong>expected</strong> error as part of a <code>CmdResponse</code> value, which is not a Rust <code>Error</code>, but also, given an <strong>unexpected error</strong> with <code>Error</code>, will report it in an orderly fashion.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-storage z-type z-rust">const</span> <span class="z-constant z-other z-rust">DEFAULT_ERR_EXIT_CODE</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-storage z-type z-rust">i32</span> <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-constant z-numeric z-integer z-decimal z-rust">1</span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">result_exit</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">res</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>CmdResponse<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">let</span> exit_with <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-keyword z-control z-rust">match</span> res <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>cmd</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> <span class="z-keyword z-operator z-rust">=&gt;</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-control z-rust">if</span> <span class="z-storage z-type z-rust">let</span> <span class="z-support z-type z-rust">Some</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>message</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> <span class="z-keyword z-operator z-assignment z-rust">=</span> cmd<span class="z-punctuation z-accessor z-dot z-rust">.</span>message <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-control z-rust">if</span> <span class="z-meta z-path z-rust">exitcode<span class="z-punctuation z-accessor z-rust">::</span></span>is_success<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>cmd<span class="z-punctuation z-accessor z-dot z-rust">.</span>code</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">eprintln!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span><span class="z-constant z-other z-placeholder z-rust">{message}</span><span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> <span class="z-keyword z-control z-rust">else</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">eprintln!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span><span class="z-constant z-other z-placeholder z-rust">{}</span> <span class="z-constant z-other z-placeholder z-rust">{}</span><span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-separator z-rust">,</span> <span class="z-support z-function z-rust">style</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>error:<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">red</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">bold</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> <span class="z-support z-function z-rust">style</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>message</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">red</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> cmd<span class="z-punctuation z-accessor z-dot z-rust">.</span>code </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Err</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>e</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> <span class="z-keyword z-operator z-rust">=&gt;</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">eprintln!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>error: <span class="z-constant z-other z-placeholder z-rust">{e:?}</span><span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-constant z-other z-rust">DEFAULT_ERR_EXIT_CODE</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-function z-rust">exit</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>exit_with</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h2 id="think-about-your-surface-area">Think about your surface area</h2> <p><img src="https://rng0.io/blog/errors-in-rust-a-formula/3.png" alt="" /></p> <p>Make sure your crate exposes a <em>single</em> error type.</p> <p>Why?</p> <ul> <li><strong>It will make your users be able to create a single</strong> <code>from</code> <strong>conversion</strong>, and be done with it</li> <li>Documentation of <strong>what to handle and how to handle</strong> is in one single place</li> <li>For those who want <strong>to cover <em>all</em> cases</strong>, matching all variants of a single <code>Error</code> enum covers it faithfull</li> <li><strong>Reporting, debugging, and operability</strong> focus — while working with your crate, users are getting to know a single error type and its context intimately and so be able to handle it effectively whether within a debugging session or through code</li> <li><strong>One error type means one kind of</strong> <code>Result</code> <strong>type</strong>, which is by itself a better API design</li> <li>If needed, nest other error types <strong>within that single error type in one of its variant</strong></li> </ul> <h2 id="errors-a-formula">Errors: a Formula</h2> <p>Follow these steps for error nirvana in Rust.</p> <h3 id="1-add-and-learn-your-dependencies">1. Add and learn your dependencies</h3> <ul> <li><code>thiserror</code> for all errors in your crate</li> <li><code>eyre</code> for CLIs</li> </ul> <h3 id="2-create-a-base-error-type-per-crate">2. Create a base error type per crate</h3> <p>You can place it in your top level <code>mod.rs</code> or <code>lib.rs</code>.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> lib/mod.rs </span></span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">derive</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">thiserror::Error<span class="z-punctuation z-separator z-rust">,</span> Debug</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-enum z-rust">enum</span> <span class="z-entity z-name z-enum z-rust">Error</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">error</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>{0}<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> Message<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">String</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> a central IO wrapper </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">error</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">transparent</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-constant z-other z-rust">IO</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">from</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> <span class="z-meta z-path z-rust">std<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">io<span class="z-punctuation z-accessor z-rust">::</span></span>Error</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> will be used with `.map_err(Box::from)?;` </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">error</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">transparent</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> Any<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">from</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn <span class="z-meta z-path z-rust">std<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">error<span class="z-punctuation z-accessor z-rust">::</span></span>Error <span class="z-keyword z-operator z-rust">+</span> <span class="z-support z-type z-rust">Send</span> <span class="z-keyword z-operator z-rust">+</span> <span class="z-support z-type z-rust">Sync</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> some other common conversions </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">error</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">transparent</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> Pattern<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">from</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> <span class="z-meta z-path z-rust">regex<span class="z-punctuation z-accessor z-rust">::</span></span>Error</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">error</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">transparent</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> Json<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">from</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> <span class="z-meta z-path z-rust">serde_json<span class="z-punctuation z-accessor z-rust">::</span></span>Error</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h3 id="3-match-map-wrap">3. Match, map, wrap</h3> <p>Take errors from dependencies, stdlib or your other modules, and where needed <code>match</code> and extract information, or map them with <code>map_err</code>, or wrap them into your own error type.</p> <h3 id="4-in-your-code-keep-using-conversions">4. In your code, keep using <code>?</code> conversions</h3> <p>Let the compiler help you and make automatic <code>from</code> conversions.</p> <p>If an error from a 3rd party library cannot be automatically converted, add an <code>enum</code> variant to your top level crate error with the <code>#[from]</code> attribute. Verify that you're not creating competing variants (the compiler will let you know).</p> <p>When trying to convert multiple layers of errors, code your own <code>From</code> trait to help centralize error making decision points.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">From</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>RustBertError<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span> <span class="z-keyword z-other z-rust">for</span></span><span class="z-meta z-impl z-rust"> <span class="z-entity z-name z-impl z-rust">Error</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">from</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">e</span><span class="z-punctuation z-separator z-rust">:</span> RustBertError</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-storage z-type z-rust">Self</span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">Self</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>EmbeddingErr<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust"><span class="z-keyword z-other z-rust">super</span><span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">EmbeddingError<span class="z-punctuation z-accessor z-rust">::</span></span>SentenceEmbedding<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">Box</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>from<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>e</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Remember, you can also <code>.map_err</code> into an error that can be converted via <code>?</code> and <code>from</code> traits.</p> <h3 id="5-create-contextful-variants-but-don-t-overdo-it">5. Create contextful variants but don’t overdo it</h3> <p>Create variants that contain information that is available when an error is created.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust">InvalidSyntax<span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> file<span class="z-punctuation z-separator z-rust">:</span> <span class="z-support z-type z-rust">String</span><span class="z-punctuation z-separator z-rust">,</span> line<span class="z-punctuation z-separator z-rust">:</span> <span class="z-storage z-type z-rust">usize</span><span class="z-punctuation z-separator z-rust">,</span> reason<span class="z-punctuation z-separator z-rust">:</span> <span class="z-support z-type z-rust">String</span> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></code></pre> <p>But don’t overdo it by containing every single piece of information there is in the universe.</p> <h3 id="6-think-about-operability">6. Think about operability</h3> <p>Ask yourself:</p> <ul> <li>Can a user <strong>do something</strong> to recover with the information you’re encoding with an error you’re creating?</li> <li>Is it an <strong>automatic recovery? or manual?</strong></li> <li>Is there an importance for <strong>time? for space? for resources? hardware?</strong></li> <li>Will errors <strong>appear in logs?</strong> what would they need to contain?</li> <li>After crashing, will an error give a user <strong>enough information to fix an issue</strong>?</li> </ul> <h3 id="7-when-all-else-fails-box-it">7. When all else fails, <code>Box</code> it</h3> <p>Use a variant in your error that can just take a <code>dyn Error</code> if the originating error is not important but you have to create an ergonomic codebase.</p> <p>🙋‍♀️ Have questions? want to say hi? <a href="https://twitter.com/jondot">follow me on Twitter</a></p> Rust and LLM AI Infrastructure: Embracing the Power of Performance Wed, 02 Aug 2023 00:00:00 +0000 Unknown https://rng0.io/blog/rust-and-llm-ai-infrastructure-embracing-the-power-of-performance/ https://rng0.io/blog/rust-and-llm-ai-infrastructure-embracing-the-power-of-performance/ <p><img src="https://rng0.io/blog/rust-and-llm-ai-infrastructure-embracing-the-power-of-performance/1.png" alt="" /></p> <p>Building an LLM or AI infrastructure in Rust can offer several benefits despite Python’s dominance in the AI space.</p> <ol> <li>Performance: Rust is known for its high performance and low-level control, which can be crucial for building large-scale AI systems. Language models, especially deep learning models, can be computationally intensive. Rust’s performance can lead to significant speed improvements compared to Python, making it more suitable for efficiently handling computationally expensive tasks.</li> <li>Memory Safety: Rust’s strict compiler rules and ownership model ensure memory safety, preventing common bugs like null pointer dereferences and data races. This can make Rust-based AI systems more reliable and less prone to crashes, which is particularly important for long-running language models or critical AI applications.</li> <li>Concurrency: Rust’s built-in support for concurrency and lightweight threads can lead to the efficient utilization of multicore processors. This can be valuable when implementing parallel processing for training large language models or handling multiple inference requests simultaneously.</li> </ol> <p>Yes, it is worth remembering that Python’s extensive ecosystem, well-established libraries (e.g., TensorFlow, PyTorch), and ease of use have made it the go-to choice for many AI projects. But if you have a sense of exploration and pinned out your specific requirements and Rust is there at the top, this might be a relevant read for you.</p> <p>Let’s try to answer the question: does Rust have a go-to all-inclusive library like Python has <a href="https://langchain-langchain.vercel.app/">langchain</a> for working with LLMs?</p> <blockquote> <p>P.S. It might be worthwhile to exercise some of the building blocks behind langchain in Python. For that, I created <a href="https://github.com/jondot/langchain-llm-katas">langchain-llm-katas</a>, give it a try.</p> </blockquote> <h2 id="chain-library-building-blocks">Chain Library Building Blocks</h2> <p>If we zoom out for a minute and ignore high-level workflows such as agents, toolkits, and baked use cases, we’ll be looking at the common infrastructure beneath all of those which allow for these high-level constructs to exist:</p> <ul> <li>LLM — loading, calling LLMs, and supporting different models, which need:</li> <li>Embedding and Tokenization — turning text into embeddings, where text is loaded with:</li> <li>Loaders — turning various document formats into LLM digestible, flat text, which needs to be post-processed by:</li> <li>Splitters — to create legible, useful chunks of the original text, to work with LLM token limits and be stored in:</li> <li>Vector databases — which are used to formulate prompts that are sent to LLMs and also for powering:</li> <li>Memory — which is done to support context and sessions with LLMs. But also, we have the infrastructure for:</li> <li>Templates — that structure a request from an LLM, which may contain a call for:</li> <li>Tools — which are a collection of real-world tools such as a calculator or a browser for the LLM to automate by reading a structured prompt</li> </ul> <p>So what does all of that look like currently in Rust?</p> <h2 id="infrastructure-and-building-blocks">Infrastructure and building blocks</h2> <h2 id="llm-and-transformers">LLM and Transformers</h2> <h3 id="llm"><a href="https://github.com/rustformers/llm">llm</a></h3> <p><code>llm</code> is an ecosystem of Rust libraries for working with large language models. It's built on top of the fast, efficient <a href="https://github.com/rustformers/llm/blob/main/crates/ggml">GGML</a> library for machine learning. It is a solid, expansive, and stable library for working with models, to the point that this is the one library to always pick, and it’s a good thing that there aren’t many alternatives.</p> <p>Here’s how to run inference using just the <code>llm</code> crate:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> model <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">llm<span class="z-punctuation z-accessor z-rust">::</span></span>load_dynamic<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-support z-type z-rust">Some</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>model_architecture</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-keyword z-operator z-bitwise z-rust">&amp;</span>model_path<span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> tokenizer_source<span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-support z-type z-rust">Default</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>default<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-meta z-path z-rust">llm<span class="z-punctuation z-accessor z-rust">::</span></span>load_progress_callback_stdout<span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span><span class="z-source z-rust"><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">unwrap_or_else</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">|</span></span></span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-variable z-parameter z-rust">err</span><span class="z-punctuation z-section z-parameters z-end z-rust">|</span></span> </span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">panic!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>Failed to load {model_architecture} model from {model_path:?}: {err}<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> <span class="z-storage z-modifier z-rust">mut</span> session <span class="z-keyword z-operator z-assignment z-rust">=</span> model<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">start_session</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">Default</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>default<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> res <span class="z-keyword z-operator z-assignment z-rust">=</span> session<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-meta z-path z-rust">infer<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-generic z-rust"><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>Infallible<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> model<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">as_ref</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-keyword z-operator z-bitwise z-rust">&amp;</span><span class="z-storage z-modifier z-rust">mut</span> <span class="z-meta z-path z-rust">rand<span class="z-punctuation z-accessor z-rust">::</span></span>thread_rng<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-keyword z-operator z-bitwise z-rust">&amp;</span><span class="z-meta z-path z-rust">llm<span class="z-punctuation z-accessor z-rust">::</span></span>InferenceRequest <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> prompt<span class="z-punctuation z-separator z-rust">:</span> prompt<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">into</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> parameters<span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-bitwise z-rust">&amp;</span><span class="z-meta z-path z-rust">llm<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">InferenceParameters<span class="z-punctuation z-accessor z-rust">::</span></span>default<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> play_back_previous_tokens<span class="z-punctuation z-separator z-rust">:</span> <span class="z-constant z-language z-rust">false</span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> maximum_token_count<span class="z-punctuation z-separator z-rust">:</span> <span class="z-support z-type z-rust">None</span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> OutputRequest </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-keyword z-operator z-bitwise z-rust">&amp;</span><span class="z-storage z-modifier z-rust">mut</span> <span class="z-support z-type z-rust">Default</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>default<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">|</span></span></span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-variable z-parameter z-rust">r</span><span class="z-punctuation z-section z-parameters z-end z-rust">|</span></span> </span><span class="z-meta z-function z-closure z-rust"><span class="z-keyword z-control z-rust">match</span> r <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-path z-rust">llm<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">InferenceResponse<span class="z-punctuation z-accessor z-rust">::</span></span>PromptToken<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>t</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> <span class="z-keyword z-operator z-bitwise z-rust">|</span> <span class="z-meta z-path z-rust">llm<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">InferenceResponse<span class="z-punctuation z-accessor z-rust">::</span></span>InferredToken<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>t</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> <span class="z-keyword z-operator z-rust">=&gt;</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">print!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span><span class="z-constant z-other z-placeholder z-rust">{t}</span><span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-path z-rust">std<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">io<span class="z-punctuation z-accessor z-rust">::</span></span>stdout<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">flush</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">unwrap</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">llm<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">InferenceFeedback<span class="z-punctuation z-accessor z-rust">::</span></span>Continue</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-operator z-rust">_</span> <span class="z-keyword z-operator z-rust">=&gt;</span> <span class="z-support z-type z-rust">Ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">llm<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">InferenceFeedback<span class="z-punctuation z-accessor z-rust">::</span></span>Continue</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <h3 id="rust-bert"><a href="https://github.com/guillaume-be/rust-bert"><strong>rust-bert</strong></a></h3> <p>Probably the most versatile and productive library out there for rust for using transformer models, inspired by <code>huggingface</code> s transformers library. This is a one-stop shop for local transformer models for different tasks, from translation to embedding.</p> <ul> <li>Lots of use cases and examples</li> <li>Very wide, as is the original transformers library</li> <li>You can get a lot done just by using this library, and you might not need anything else</li> </ul> <p>In terms of use cases, here’s an example sentiment analysis:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> sentiment_classifier <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">SentimentModel<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">Default</span><span class="z-meta z-path z-rust"><span class="z-punctuation z-accessor z-rust">::</span></span>default<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"> </span><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> input <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">[</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>Probably my all-time favorite movie, a story of selflessness, sacrifice and dedication to a noble cause, but it&#39;s not preachy or boring.<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>This film tried to be too many things all at once: stinging political satire, Hollywood blockbuster, sappy romantic comedy, family values promo...<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>If you like original gut wrenching laughter you will like this movie. If you are young or old then you will love this movie, hell even my mom liked it.<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">]</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> output <span class="z-keyword z-operator z-assignment z-rust">=</span> sentiment_classifier<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">predict</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-bitwise z-rust">&amp;</span>input</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <p>And embeddings:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> model <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">SentenceEmbeddingsBuilder<span class="z-punctuation z-accessor z-rust">::</span></span>remote<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-meta z-path z-rust">SentenceEmbeddingsModelType<span class="z-punctuation z-accessor z-rust">::</span></span>AllMiniLmL12V2 </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> </span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">create_model</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> sentences <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">[</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>this is an example sentence<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>each sentence is converted<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">]</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> output <span class="z-keyword z-operator z-assignment z-rust">=</span> model<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">encode</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-bitwise z-rust">&amp;</span>sentences</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <p>And plenty more use cases.</p> <h2 id="embeddings-and-tokenization">Embeddings and Tokenization</h2> <p>You can do embeddings with the following:</p> <ul> <li><code>rust-bert</code></li> <li><code>llm</code></li> </ul> <h3 id="tiktoken"><a href="https://github.com/zurawiki/tiktoken-rs"><strong>tiktoken</strong></a></h3> <p>This library is built on top of the OpenAI Rust tiktoken library and extends it a bit. It should be a good one-stop shop for your tokenization needs.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-keyword z-other z-rust">use</span> <span class="z-meta z-path z-rust">tiktoken_rs<span class="z-punctuation z-accessor z-rust">::</span></span>p50k_base<span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> bpe <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-support z-function z-rust">p50k_base</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">unwrap</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> tokens <span class="z-keyword z-operator z-assignment z-rust">=</span> bpe<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">encode_with_special_tokens</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>This is a sentence with spaces<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>Token count: <span class="z-constant z-other z-placeholder z-rust">{}</span><span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-separator z-rust">,</span> tokens<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">len</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <h2 id="loaders">Loaders</h2> <p>At the moment of writing, there is no unified loader like <code>unstructured</code>, that can load and convert documents without caring so much for the implementation of the specific provider. Some of these may need some elbow grease to get content into a flat, LLM-document-like format (plain text, pages), for example, looping over and extracting worksheets from Excel files.</p> <p>However, here’s a sensible mapping from file format to an appropriate Rust library:</p> <ul> <li><a href="https://lib.rs/crates/csv">csv</a></li> <li>email: <a href="https://crates.io/crates/eml-parser">eml, msg</a></li> <li>epub <a href="https://lib.rs/crates/pandoc">pandoc</a></li> <li><a href="https://lib.rs/crates/calamine">xls</a></li> <li><a href="https://lib.rs/crates/html2text">html</a> <a href="https://lib.rs/crates/pandoc">pandoc</a></li> <li><a href="https://github.com/antimatter15/tesseract-rs">images (ocr)</a></li> <li><a href="https://lib.rs/crates/pulldown-cmark%20pandoc">md</a></li> <li><a href="https://github.com/hydrobeam/org-rust">org mode (.org)</a></li> <li><a href="https://lib.rs/crates/spreadsheet-ods">open office</a> <a href="https://lib.rs/crates/pandoc">pandoc</a></li> <li><a href="https://lib.rs/crates/pdf-extract">pdf</a></li> <li>txt (no need)</li> <li><a href="https://github.com/anvie/dotext">pptx</a> <a href="https://lib.rs/crates/pandoc">pandoc</a></li> <li><a href="https://github.com/anvie/dotext">docx</a> <a href="https://lib.rs/crates/pandoc">pandoc</a></li> <li><a href="https://crates.io/crates/rst">rst</a> <a href="https://lib.rs/crates/pandoc">pandoc</a></li> <li>rtf <a href="https://lib.rs/crates/pandoc">pandoc</a></li> <li><a href="https://crates.io/crates/xml-rs">xml</a></li> </ul> <h2 id="splitters">Splitters</h2> <p><a href="https://github.com/benbrandt/text-splitter">https://github.com/benbrandt/text-splitter</a></p> <p>The only library that is practical enough for splitting. Seems to be taking a “split properly” approach where you don’t need to choose if to split by newlines, characters, recursively, or tokens. It will descend and use an appropriate method to maximize chunk sizes.</p> <h2 id="prompts">Prompts</h2> <p>At the moment of writing, there’s no general consensus and no library that unifies the template concept like langchain has. However, taking a templating library and building on top of it is a great way to implement prompts.</p> <p>Rust has a few fantastic templating libraries which suit templating:</p> <ul> <li><a href="https://lib.rs/crates/handlebars">Handlebars</a> — for standard, familiar, minimal logic templates</li> <li><a href="https://docs.rs/tera/latest/tera/">Tera</a> — for those familiar with jinja2</li> <li><a href="https://lib.rs/crates/liquid">Liquid</a> — for those familiar with liquid</li> </ul> <p>Most times, handlebars is the best bet that’ll be familiar to a general audience.</p> <h2 id="vector-databases">Vector Databases</h2> <p>Currently, there’s no unified interface for vector databases that provides a general wrapper around many providers. Survey a few open source projects around LLMs. You’ll find reimplementations of the same interfaces over and over, which is a pity, and still, no one library provides a generic interface.</p> <p>Given such a simplistic interface, there is a good ROI in building such a library. It should basically be:</p> <ul> <li><code>add_documents</code></li> <li><code>similarity_search</code></li> </ul> <p>FYI: Many individual vector databases are implemented in Rust, both commercial and open source. <a href="https://github.com/qdrant/qdrant">Qdrant</a> is probably the most popular open source one.</p> <h2 id="memory">Memory</h2> <p>There are a few projects implementing memory or history. Most of them implement a service that stores indexes and search histories for you, and they can also be used as a reference for how to do your workflow in Rust dealing with AI.</p> <p>Some interesting ones are:</p> <ul> <li><a href="https://github.com/spyglass-search/memex">memex</a> — part of <a href="https://github.com/spyglass-search">spyglass</a>, a good code read, and does document storage and semantic search for LLM projects.</li> <li><a href="https://github.com/diptanu/indexify">indexify</a> — long-term LLM memory</li> <li><a href="https://github.com/getmetal/motorhead">motorhead</a> — memory/information retrieval</li> </ul> <h2 id="tools">Tools</h2> <p>As of this writing, no tool/plugin libraries are dedicated to this purpose only. That is, libraries that provide a solid interface for:</p> <ul> <li>Declaring a tool</li> <li>Capabilities</li> <li>Running a tool (safely or not)</li> <li>Structuring a tool request and response</li> </ul> <p>However, those implementations can be found in the few chain implementations that Rust currently has, which are discussed below.</p> <h2 id="chains">Chains</h2> <h3 id="llmchain-rs"><a href="https://github.com/shafishlabs/llmchain-rs">llmchain-rs</a></h3> <p>A small-scope chain library. I’d say 30% of what <code>langchain-py</code> has.</p> <ul> <li>Early times for this project</li> <li>Good amount of examples</li> <li><a href="https://github.com/datafuselabs/databend">Databend</a>-focused, but generic interfaces</li> <li>Good test coverage (it’s a challenge testing LLM infra in any case)</li> </ul> <p>Let’s look at some of the abstractions:</p> <h3 id="embedding">Embedding</h3> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-trait z-rust">trait</span> <span class="z-entity z-name z-trait z-rust">Embedding</span>: Send + Sync <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">embed_query</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">input</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-storage z-type z-rust">f32</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">embed_documents</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">inputs</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>Documents</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-storage z-type z-rust">f32</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h3 id="llm-1">LLM</h3> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-trait z-rust">trait</span> <span class="z-entity z-name z-trait z-rust">LLM</span>: Send + Sync <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">embedding</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">inputs</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>EmbeddingResult<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">generate</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">input</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>GenerateResult<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">chat</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">_input</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>ChatResult<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">unimplemented!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span><span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <ul> <li>Integrations currently are: OpenAI, Azure-OpenAI, and Databend, so probably mixing data from Databend with AI was the trigger for all of this</li> </ul> <h3 id="document">Document</h3> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">derive</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">Debug<span class="z-punctuation z-separator z-rust">,</span> Clone<span class="z-punctuation z-separator z-rust">,</span> Eq<span class="z-punctuation z-separator z-rust">,</span> PartialEq</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-struct z-rust">struct</span> </span><span class="z-meta z-struct z-rust"><span class="z-entity z-name z-struct z-rust">Document</span> </span><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-modifier z-rust">pub</span> <span class="z-variable z-other z-member z-rust">path</span><span class="z-punctuation z-separator z-type z-rust">:</span> String, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-modifier z-rust">pub</span> <span class="z-variable z-other z-member z-rust">content</span><span class="z-punctuation z-separator z-type z-rust">:</span> String, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-modifier z-rust">pub</span> <span class="z-variable z-other z-member z-rust">content_md5</span><span class="z-punctuation z-separator z-type z-rust">:</span> String, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust"><span class="z-entity z-name z-impl z-rust">Document</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">create</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">path</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span>, <span class="z-variable z-parameter z-rust">content</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-storage z-type z-rust">Self</span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> Document <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> path<span class="z-punctuation z-separator z-rust">:</span> path<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">to_string</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> content<span class="z-punctuation z-separator z-rust">:</span> content<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">to_string</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> content_md5<span class="z-punctuation z-separator z-rust">:</span> <span class="z-support z-macro z-rust">format!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span><span class="z-constant z-other z-placeholder z-rust">{:x}</span><span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-separator z-rust">,</span> <span class="z-meta z-path z-rust">md5<span class="z-punctuation z-accessor z-rust">::</span></span>compute<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>content</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">tokens</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-storage z-type z-rust">usize</span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-function z-rust">chat_tokens</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-bitwise z-rust">&amp;</span><span class="z-variable z-language z-rust">self</span><span class="z-punctuation z-accessor z-dot z-rust">.</span>content</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">unwrap</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">len</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">size</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-storage z-type z-rust">usize</span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-language z-rust">self</span><span class="z-punctuation z-accessor z-dot z-rust">.</span>content<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">len</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">derive</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">Debug</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-struct z-rust">struct</span> </span><span class="z-meta z-struct z-rust"><span class="z-entity z-name z-struct z-rust">Documents</span> </span><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">documents</span><span class="z-punctuation z-separator z-type z-rust">:</span> <span class="z-meta z-generic z-rust">RwLock<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>Document<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h3 id="prompt">Prompt</h3> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-trait z-rust">trait</span> <span class="z-entity z-name z-trait z-rust">Prompt</span>: Send + Sync <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">template</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> String</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">variables</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">format</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">input_variables</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-generic z-rust">HashMap<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span>, <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <ul> <li>Simple text-based replacement of variables only</li> <li>Single level, no advanced dependent or partial prompts</li> </ul> <h3 id="vector-store">Vector store</h3> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">async_trait</span>::<span class="z-variable z-annotation z-rust">async_trait</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-trait z-rust">trait</span> <span class="z-entity z-name z-trait z-rust">VectorStore</span>: Send + Sync <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">init</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-punctuation z-section z-group z-end z-rust">)</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">add_documents</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">inputs</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>Documents</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">similarity_search</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">query</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span>, <span class="z-variable z-parameter z-rust">k</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-storage z-type z-rust">usize</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>Document<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <ul> <li>Support for <code>databend</code> as a provider only</li> <li>Missing MMR for search (like in all interfaces I’ve seen so far)</li> <li>Missing save/load abstraction (but does it even belong in a trait?)</li> </ul> <h3 id="overall">Overall</h3> <ul> <li>Relatively solid codebase and good abstractions</li> <li>Splitters (no abstraction or strategy to choose from, but this is the case with all chain implementations in Rust I’ve seen so far)</li> <li>Memory (just early times there, it feels)</li> </ul> <h2 id="llm-chain"><a href="https://github.com/sobelio/llm-chain">llm-chain</a></h2> <p>A more popular chain library, split into Rust crates, good Rust idioms, and structure. Not a full coverage of what <code>langchain-py</code> has. Maybe 30%, and still early times here, as with all other libraries.</p> <p>Let’s look at the abstractions here:</p> <h3 id="embedding-1">Embedding</h3> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">async_trait</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-trait z-rust">trait</span> <span class="z-entity z-name z-trait z-rust">Embeddings</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-type z-rust">type</span> <span class="z-entity z-name z-type z-rust">Error</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-support z-type z-rust">Send</span> <span class="z-keyword z-operator z-arithmetic z-rust">+</span> Debug <span class="z-keyword z-operator z-arithmetic z-rust">+</span> Error <span class="z-keyword z-operator z-arithmetic z-rust">+</span> EmbeddingsError<span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">embed_texts</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">texts</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-storage z-type z-rust">f32</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, <span class="z-meta z-path z-rust"><span class="z-storage z-type z-rust"><span class="z-storage z-type z-rust">Self</span><span class="z-punctuation z-accessor z-rust">::</span></span></span>Error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">embed_query</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">query</span><span class="z-punctuation z-separator z-rust">:</span> String</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-storage z-type z-rust">f32</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, <span class="z-meta z-path z-rust"><span class="z-storage z-type z-rust"><span class="z-storage z-type z-rust">Self</span><span class="z-punctuation z-accessor z-rust">::</span></span></span>Error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h3 id="vectorstore">VectorStore</h3> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">async_trait</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-trait z-rust">trait</span> <span class="z-entity z-name z-trait z-rust">VectorStore</span>&lt;E, M = EmptyMetadata&gt; </span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust">where </span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"> E: Embeddings, </span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"> M: serde::Serialize + serde::de::DeserializeOwned, </span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-type z-rust">type</span> <span class="z-entity z-name z-type z-rust">Error</span><span class="z-punctuation z-separator z-rust">:</span> Debug <span class="z-keyword z-operator z-arithmetic z-rust">+</span> Error <span class="z-keyword z-operator z-arithmetic z-rust">+</span> VectorStoreError<span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">add_texts</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">texts</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, <span class="z-meta z-path z-rust"><span class="z-storage z-type z-rust"><span class="z-storage z-type z-rust">Self</span><span class="z-punctuation z-accessor z-rust">::</span></span></span>Error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">add_documents</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">documents</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust">Document<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>M<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, <span class="z-meta z-path z-rust"><span class="z-storage z-type z-rust"><span class="z-storage z-type z-rust">Self</span><span class="z-punctuation z-accessor z-rust">::</span></span></span>Error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">similarity_search</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"> <span class="z-variable z-parameter z-rust">query</span><span class="z-punctuation z-separator z-rust">:</span> String, </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"> <span class="z-variable z-parameter z-rust">limit</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-storage z-type z-rust">u32</span>, </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"> </span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust">Document<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>M<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, <span class="z-meta z-path z-rust"><span class="z-storage z-type z-rust"><span class="z-storage z-type z-rust">Self</span><span class="z-punctuation z-accessor z-rust">::</span></span></span>Error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h3 id="splitters-1">Splitters</h3> <p>Looks like a splitter is a <code>Tokenizer</code></p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-trait z-rust">trait</span> <span class="z-entity z-name z-trait z-rust">Tokenizer</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">tokenize_str</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">doc</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>TokenCollection, TokenizerError<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">to_string</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">tokens</span><span class="z-punctuation z-separator z-rust">:</span> TokenCollection</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span>, TokenizerError<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">split_text</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"> <span class="z-variable z-parameter z-rust">doc</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span>, </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"> <span class="z-variable z-parameter z-rust">max_tokens_per_chunk</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-storage z-type z-rust">usize</span>, </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"> <span class="z-variable z-parameter z-rust">chunk_overlap</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-storage z-type z-rust">usize</span>, </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"> </span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, TokenizerError<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h3 id="prompt-1">Prompt</h3> <p><a href="https://github.com/sobelio/llm-chain/blob/main/crates/llm-chain/src/prompt/model.rs">Prompt</a> is a construct that allows operating with an abstraction of <code>Chat</code> or <code>Text</code> messages. Templating is powerful and based on <code>tera</code>.</p> <h3 id="document-1">Document</h3> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">derive</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">Debug</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-struct z-rust">struct</span> </span><span class="z-meta z-struct z-rust"><span class="z-meta z-generic z-rust"><span class="z-entity z-name z-struct z-rust">Document</span><span class="z-meta z-generic z-rust"><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>M = EmptyMetadata<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-meta z-struct z-rust"> </span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust">where </span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"> M: serde::Serialize + serde::de::DeserializeOwned, </span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"></span><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-modifier z-rust">pub</span> <span class="z-variable z-other z-member z-rust">page_content</span><span class="z-punctuation z-separator z-type z-rust">:</span> String, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-modifier z-rust">pub</span> <span class="z-variable z-other z-member z-rust">metadata</span><span class="z-punctuation z-separator z-type z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Option</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>M<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>More similar than different in comparison to the <code>langchain-py</code> document.</p> <h3 id="documentstore-loader">DocumentStore (Loader?)</h3> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">async_trait</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-trait z-rust">trait</span> <span class="z-entity z-name z-trait z-rust">DocumentStore</span>&lt;T, M&gt; </span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust">where </span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"> T: Send + Sync, </span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"> M: Serialize + DeserializeOwned + Send + Sync, </span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-type z-rust">type</span> <span class="z-entity z-name z-type z-rust">Error</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-path z-rust">std<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">fmt<span class="z-punctuation z-accessor z-rust">::</span></span>Debug <span class="z-keyword z-operator z-arithmetic z-rust">+</span> <span class="z-meta z-path z-rust">std<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">error<span class="z-punctuation z-accessor z-rust">::</span></span>Error <span class="z-keyword z-operator z-arithmetic z-rust">+</span> DocumentStoreError<span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust">async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">get</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">id</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>T</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Option</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust">Document<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>M<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, <span class="z-meta z-path z-rust"><span class="z-storage z-type z-rust"><span class="z-storage z-type z-rust">Self</span><span class="z-punctuation z-accessor z-rust">::</span></span></span>Error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">next_id</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>T, <span class="z-meta z-path z-rust"><span class="z-storage z-type z-rust"><span class="z-storage z-type z-rust">Self</span><span class="z-punctuation z-accessor z-rust">::</span></span></span>Error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">insert</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-modifier z-rust">mut</span> <span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">documents</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-meta z-generic z-rust">HashMap<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>T, <span class="z-meta z-generic z-rust">Document<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>M<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-punctuation z-section z-group z-end z-rust">)</span>, <span class="z-meta z-path z-rust"><span class="z-storage z-type z-rust"><span class="z-storage z-type z-rust">Self</span><span class="z-punctuation z-accessor z-rust">::</span></span></span>Error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h3 id="overall-1">Overall</h3> <ul> <li>Good separation between features and crates</li> <li>Feels rusty</li> <li>Lacking splitters, loaders</li> <li>Lacking integrations generally, and especially vector stores</li> </ul> <h2 id="smartgpt"><a href="https://github.com/Cormanz/smartgpt">SmartGPT</a></h2> <p>This is not a library nor infrastructure, but the internals are built well and can be of good use for a chain library. Let’s have a look at the abstractions:</p> <h3 id="llm-2">LLM</h3> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-trait z-rust">trait</span> <span class="z-entity z-name z-trait z-rust">LLMModel</span> : Send + Sync <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">get_response</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">messages</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>[Message], <span class="z-variable z-parameter z-rust">max_tokens</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Option</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-storage z-type z-rust">u16</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, <span class="z-variable z-parameter z-rust">temperature</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Option</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-storage z-type z-rust">f32</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span>, <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn Error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">get_base_embed</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">text</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-storage z-type z-rust">f32</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn Error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">get_token_count</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">text</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>[Message]</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-storage z-type z-rust">usize</span>, <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn Error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">get_token_limit</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-storage z-type z-rust">usize</span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">get_tokens_from_text</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">text</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn Error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Some methods were removed because they were default implemented.</p> <h3 id="message">Message</h3> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">derive</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">Clone<span class="z-punctuation z-separator z-rust">,</span> Debug</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-enum z-rust">enum</span> <span class="z-entity z-name z-enum z-rust">Message</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> User<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">String</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> Assistant<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">String</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"> System<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-type z-rust">String</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-enum z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Modeling a message on a User-AI-System pattern</p> <h3 id="memorysystem-history">MemorySystem (History?)</h3> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-trait z-rust">trait</span> <span class="z-entity z-name z-trait z-rust">MemorySystem</span> : Send + Sync <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">store_memory</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-modifier z-rust">mut</span> <span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">llm</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>LLM, <span class="z-variable z-parameter z-rust">memory</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-punctuation z-section z-group z-end z-rust">)</span>, <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn Error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust">async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">get_memory_pool</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-modifier z-rust">mut</span> <span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">llm</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>LLM, <span class="z-variable z-parameter z-rust">memory</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span>, <span class="z-variable z-parameter z-rust">min_count</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-storage z-type z-rust">usize</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>RelevantMemory<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn Error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">get_memories</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"> } </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust">... </span></span></span></span></span></code></pre> <p>A good interface of memory</p> <h3 id="command-tool">Command (Tool?)</h3> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-trait z-rust">trait</span> <span class="z-entity z-name z-trait z-rust">CommandImpl</span> : Send + Sync <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"> async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">invoke</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span>, <span class="z-variable z-parameter z-rust">ctx</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-modifier z-rust">mut</span> CommandContext, <span class="z-variable z-parameter z-rust">args</span><span class="z-punctuation z-separator z-rust">:</span> ScriptValue</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>CommandResult, <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn Error<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">box_clone</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">&amp;</span><span class="z-variable z-parameter z-rust">self</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Box</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>dyn CommandImpl<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-trait z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h3 id="overall-2">Overall</h3> <ul> <li>This project has a relatively rich tool collection and might be the best start for a generic library that implements tools</li> <li>Codebase is very pragmatic because almost everything is actually in use</li> <li>Where functionality was not needed — it does not exist (naturally), and this is where the codebase lacks, in comparison to langchain: splitters, loaders, prompt templates, vector stores, and a general sense of multiple providers</li> <li>To use it as a general-purpose langchain library would be awkward unless you hit the use cases that this project solves already. Which I think will not be enough pretty quickly.</li> </ul> <p>Thanks for reading.</p> Building CLI Apps in Rust — What You Should Consider Thu, 27 Apr 2023 00:00:00 +0000 Unknown https://rng0.io/blog/building-cli-apps-in-rust-what-you-should-consider/ https://rng0.io/blog/building-cli-apps-in-rust-what-you-should-consider/ <h3 id="this-is-part-guide-part-post-about-what-you-should-consider-before-building-a-new-cli-app-in-rust">This is part guide, part post about what you should consider before building a new CLI app in Rust.</h3> <p><img src="https://rng0.io/blog/building-cli-apps-in-rust-what-you-should-consider/1.jpg" alt="" /></p> <p>Photo by <a href="https://unsplash.com/@mmintel?utm_source=medium&amp;utm_medium=referral">Marc Mintel</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></p> <blockquote> <p><strong>Like this article? have questions?</strong> <a href="https://twitter.com/jondot"><strong>Follow me on twitter</strong></a> <strong>and ask away:)</strong></p> </blockquote> <p>Before jumping into a <code>cargo new</code> and hitting it, read up and see how to boost your development experience, CLI ergonomics, and project maintainability.</p> <h2 id="structure">Structure</h2> <p>When you have a CLI app, you have a bunch of flags and commands, and for these the suitable logical module.</p> <p>For example, for a <code>git clone</code> subcommand you have some <em>clone</em> functionality, but for a <code>git commit</code> subcommand you might have a completely different functionality that can live in a completely different module (and should). So, this can be a different module and also a different <em>file</em>.</p> <p>Or, you can have a simple flat structure of a CLI app that does just one thing but takes various flags as tweaks to that one single thing:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> demo<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>help</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">A</span></span><span class="z-meta z-function-call z-arguments z-shell"> simple to use, efficient, and full-featured Command Line Argument Parser</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Usage:</span></span><span class="z-meta z-function-call z-arguments z-shell"> demo<span class="z-keyword z-control z-regexp z-set z-begin z-shell">[</span>EXE<span class="z-keyword z-control z-regexp z-set z-end z-shell">]</span> <span class="z-keyword z-control z-regexp z-set z-begin z-shell">[</span>OPTIONS<span class="z-keyword z-control z-regexp z-set z-end z-shell">]</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>name</span> <span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>NAME<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> </span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Options:</span></span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">-n,</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>name</span> <span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>NAME<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> Name of the person to greet</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">-c,</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>count</span> <span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>COUNT<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> Number of times to greet <span class="z-keyword z-control z-regexp z-set z-begin z-shell">[</span>default: 1<span class="z-keyword z-control z-regexp z-set z-end z-shell">]</span></span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">-h,</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>help</span> Print help</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">-V,</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>version</span> Print version</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> demo<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>name</span> Me</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Hello</span></span><span class="z-meta z-function-call z-arguments z-shell"> Me!</span> </span></code></pre> <p>So when we talk about a command layout it should mean splitting to files, and then we also want to talk about file structure and project layout.</p> <p>After reviewing a good number of popular CLI apps in Rust I found that there are generally three kinds of app structures:</p> <ul> <li>Ad-hoc, see <code>xh</code> as an example, with any folder structure</li> <li>Flat, with a folder structure such as:</li> </ul> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">src/ </span><span class="z-text z-plain"> main.rs </span><span class="z-text z-plain"> clone.rs </span><span class="z-text z-plain"> .. </span></code></pre> <ul> <li><strong>Sub-commands</strong>, where the structure is nested, here’s <code>delta</code> as an example, with a folder structure such as</li> </ul> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">src/ </span><span class="z-text z-plain"> cmd/ </span><span class="z-text z-plain"> clone.rs </span><span class="z-text z-plain"> main.rs </span></code></pre> <p>You can go with a flat or nested structure using my starter project: <code>rust-starter</code> and use what you need, delete what you don’t need.</p> <p>While you’re at it, it’s always good to split the core of your app and its interface. I find a good rule of thumb is to think about creating:</p> <ul> <li>A library</li> <li>A CLI that uses this library</li> <li>And something that helps generalize and solidify the API of this library: some other GUI (that will never exist) that might use this library</li> </ul> <p>More often than not, especially in Rust, I find that this split was extremely useful and the use-cases happen to present themselves to me to use the library as a library by its own.</p> <p>Typically it splits to three parts:</p> <ol> <li>A library</li> <li>A main workflow/runner, (e.g. <code>workflow.rs</code>) that is driving the CLI app</li> <li>The CLI parts (prompting, exit handling, parsing flags, and commands) that maps into the workflow</li> </ol> <h2 id="flags-options-arguments-and-commands">Flags, Options, Arguments, and Commands</h2> <p>Even if we rule out CLI apps that are very graphical like Vim (uses TUI, etc.), we still have to address UI/UX concerns in our CLI, in the shape of flags, commands, and arguments that a program receives.</p> <p>There are several different ways to specify and parse command line options, each with its own set of conventions and best practices.</p> <p>According to the <a href="https://en.wikipedia.org/wiki/POSIX">POSIX</a> standard, options are specified using a single dash followed by a single letter, and can be combined into a single argument (e.g. <code>-abc</code>). Long options, which are usually more descriptive and easier to read, are specified using two dashes followed by a word (e.g. <code>--option</code>). Options can also accept values, which are specified using an equals sign (e.g. <code>-o=value</code> or <code>--option=value</code>).</p> <p>Positional arguments are the arguments that a command line application expects to be specified in a particular order. They are not prefixed with a dash, and are often used to specify required data that the application needs to function properly.</p> <p>The POSIX standard also defines a number of special options, such as <code>-h</code> or <code>--help</code> for displaying a help message, and <code>-v</code> or <code>--verbose</code> for verbose output. These options are widely recognized and used by many command line applications, making it easier for users to discover and use different features.</p> <p>Overall, the POSIX standard provides a set of conventions for specifying and parsing command line options that are widely recognized and followed by many command line applications, making it easier for users to understand and use different command line tools.</p> <p>Or in other words, when designing the interface for a command-line app, we need to think about:</p> <ol> <li><strong>Sub-commands:</strong> Some command line applications allow users to specify sub-commands, which are essentially additional sub-applications that can be run within the main application. For example, the <code>git</code> command allows users to specify sub-commands such as <code>commit</code>, <code>push</code>, and <code>pull</code>. Sub-commands are often used to group related functionality within a single application and make it easier for users to discover and use different features.</li> <li>Positional arguments: Positional arguments are the arguments that a command line application expects to be specified in a particular order. For example, the <code>cp</code> command expects two positional arguments: the source file and the destination file. Positional arguments are often used to specify required data that the application needs to function properly.</li> <li><strong>Flags/options</strong>: Flags are command line options that do not expect a value to be specified. They are often used to toggle a particular behavior or setting in the application. Flags are typically specified using a single dash followed by a single letter (e.g. <code>-v</code> for verbose output) or a double dash followed by a word (e.g. <code>--verbose</code> for verbose output). Flags can also accept optional values, which are specified using an equals sign (e.g. <code>--output=file.txt</code>).</li> </ol> <p>To get a great library to rely on, that will get you from a simple app to a complex one without the need for migrating to another library, you can use <a href="https://github.com/clap-rs/clap">the clap crate</a>. Clap will carry you a long way; consider using an alternative only if you have a special requirement such as faster compile time, smaller binary, or similar.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-path z-rust">Command<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>git<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span><span class="z-source z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">about</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>A fictional versioning CLI<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span><span class="z-source z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">subcommand_required</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-constant z-language z-rust">true</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span><span class="z-source z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">arg_required_else_help</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-constant z-language z-rust">true</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span><span class="z-source z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">allow_external_subcommands</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-constant z-language z-rust">true</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span><span class="z-source z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">subcommand</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-meta z-path z-rust">Command<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>clone<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">about</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>Clones repos<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">arg</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-macro z-rust">arg!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-comparison z-rust">&lt;</span><span class="z-constant z-other z-rust">REMOTE</span><span class="z-keyword z-operator z-comparison z-rust">&gt;</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>The remote to clone<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">arg_required_else_help</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-constant z-language z-rust">true</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> </span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span><span class="z-source z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">subcommand</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-meta z-path z-rust">Command<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>diff<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">about</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>Compare two commits<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">arg</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-macro z-rust">arg!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>base<span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-constant z-other z-rust">COMMIT</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">arg</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-macro z-rust">arg!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>head<span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-constant z-other z-rust">COMMIT</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">arg</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-macro z-rust">arg!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>path<span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-constant z-other z-rust">PATH</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">last</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-constant z-language z-rust">true</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">arg</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-group z-rust"> <span class="z-support z-macro z-rust">arg!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-arithmetic z-rust">-</span><span class="z-keyword z-operator z-arithmetic z-rust">-</span>color <span class="z-keyword z-operator z-comparison z-rust">&lt;</span><span class="z-constant z-other z-rust">WHEN</span><span class="z-keyword z-operator z-comparison z-rust">&gt;</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">value_parser</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>always<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>auto<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>never<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">num_args</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-constant z-numeric z-integer z-decimal z-rust">0</span><span class="z-keyword z-operator z-range z-rust">..</span><span class="z-keyword z-operator z-assignment z-rust">=</span><span class="z-constant z-numeric z-integer z-decimal z-rust">1</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">require_equals</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-constant z-language z-rust">true</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">default_value</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>auto<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">default_missing_value</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>always<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-group z-rust"> </span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> </span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span><span class="z-source z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">subcommand</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-meta z-path z-rust">Command<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>push<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">about</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>pushes things<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">arg</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-macro z-rust">arg!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-comparison z-rust">&lt;</span><span class="z-constant z-other z-rust">REMOTE</span><span class="z-keyword z-operator z-comparison z-rust">&gt;</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>The remote to target<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">arg_required_else_help</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-constant z-language z-rust">true</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> </span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span><span class="z-source z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">subcommand</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-meta z-path z-rust">Command<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>add<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">about</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>adds things<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">arg_required_else_help</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-constant z-language z-rust">true</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">arg</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-macro z-rust">arg!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-comparison z-rust">&lt;</span><span class="z-constant z-other z-rust">PATH</span><span class="z-keyword z-operator z-comparison z-rust">&gt;</span> <span class="z-keyword z-operator z-range z-rust">...</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>Stuff to add<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">value_parser</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">clap<span class="z-punctuation z-accessor z-rust">::</span></span>value_parser<span class="z-keyword z-operator z-logical z-rust">!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>PathBuf</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> </span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span><span class="z-source z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">subcommand</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-meta z-path z-rust">Command<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>stash<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">args_conflicts_with_subcommands</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-constant z-language z-rust">true</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">args</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-function z-rust">push_args</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">subcommand</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">Command<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>push<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">args</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-function z-rust">push_args</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">subcommand</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">Command<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>pop<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">arg</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-macro z-rust">arg!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-constant z-other z-rust">STASH</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">subcommand</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">Command<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>apply<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">arg</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-macro z-rust">arg!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-constant z-other z-rust">STASH</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"> </span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></code></pre> <p>A minimalistic alternative for clap is <a href="https://lib.rs/crates/argh"><strong>argh</strong></a>. Why choose something else?</p> <ul> <li><code>clap</code> may be too big in size for your binary, and you <em>really</em> care about binary size (we’re talking numbers like 300kb vs 50kb here)</li> <li><code>clap</code> may take longer to compile (but it’s fast enough for most people)</li> <li>You might have a very simple CLI interface and appreciate code that fits half a screen for everything you need to do.</li> </ul> <p>Here’s an example using <code>argh</code></p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">derive</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">Debug<span class="z-punctuation z-separator z-rust">,</span> FromArgs</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-storage z-type z-struct z-rust">struct</span> </span><span class="z-meta z-struct z-rust"><span class="z-entity z-name z-struct z-rust">AppArgs</span> </span><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-documentation z-rust"><span class="z-punctuation z-definition z-comment z-rust">///</span> task to run </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">argh</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">positional</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">task</span><span class="z-punctuation z-separator z-type z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Option</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-documentation z-rust"><span class="z-punctuation z-definition z-comment z-rust">///</span> list tasks </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">argh</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">switch<span class="z-punctuation z-separator z-rust">,</span> short <span class="z-keyword z-operator z-rust">=</span> &#39;l&#39;</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">list</span><span class="z-punctuation z-separator z-type z-rust">:</span> <span class="z-storage z-type z-rust">bool</span>, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-documentation z-rust"><span class="z-punctuation z-definition z-comment z-rust">///</span> root path (default &quot;.&quot;) </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">argh</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">option<span class="z-punctuation z-separator z-rust">,</span> short <span class="z-keyword z-operator z-rust">=</span> &#39;p&#39;</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">path</span><span class="z-punctuation z-separator z-type z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Option</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-documentation z-rust"><span class="z-punctuation z-definition z-comment z-rust">///</span> init local config </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">argh</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">switch<span class="z-punctuation z-separator z-rust">,</span> short <span class="z-keyword z-operator z-rust">=</span> &#39;i&#39;</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">init</span><span class="z-punctuation z-separator z-type z-rust">:</span> <span class="z-storage z-type z-rust">bool</span>, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>And then just:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> args<span class="z-punctuation z-separator z-rust">:</span> AppArgs <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">argh<span class="z-punctuation z-accessor z-rust">::</span></span>from_env<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <h2 id="configuration">Configuration</h2> <p>In most operating systems, <strong>there are standard locations for storing configuration files</strong> and other user-specific data. These locations are often referred to as “home folders” or “profile directories”, and are used to store configuration files, application data, and other user-specific data.</p> <h2 id="unix-like-systems-linux-macos">Unix-like systems (Linux / macOS)</h2> <p>The home folder is typically located at <code>/home/username</code> or <code>/Users/username</code>, and is used to store configuration files and other data that is specific to the user. The home folder is often referred to as the <code>$HOME</code> directory, and can be accessed using the <code>~</code> symbol (e.g. <code>~/.bashrc</code>).</p> <p>Within the home directory, there is often a <code>.config</code> folder (also known as the "configuration directory") that is used to store configuration files and other data that is specific to the user. The <code>.config</code> folder is a “dot” directory, so if you don’t see it, use a terminal. An application might store its configuration files in a subdirectory of the <code>.config</code> folder, such as <code>~/.config/myapp</code></p> <p>On Windows, the home folder is typically located at <code>C:\\Users\\username</code>, and is used to store configuration files and other data that is specific to the user. The home folder is often referred to as the "user profile" directory, and can be accessed using the <code>%USERPROFILE%</code> environment variable (e.g. <code>%USERPROFILE%\\AppData\\Roaming</code>).</p> <p>To hide all of this complexity, you can use <a href="https://crates.io/crates/dirs">the dirs crate.</a></p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-path z-rust">dirs<span class="z-punctuation z-accessor z-rust">::</span></span>home_dir<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> Lin: Some(/home/alice) </span></span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> Win: Some(C:\\Users\\Alice) </span></span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> Mac: Some(/Users/Alice) </span></span></code></pre> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-path z-rust">dirs<span class="z-punctuation z-accessor z-rust">::</span></span>config_dir<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> Lin: Some(/home/alice/.config) </span></span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> Win: Some(C:\\Users\\Alice\\AppData\\Roaming) </span></span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> Mac: Some(/Users/Alice/Library/Application Support) </span></span></code></pre> <h2 id="config-files">Config Files</h2> <p>Reading configuration content is easy with <code>serde</code> because it makes it a 3 stage process:</p> <ol> <li>“Shape” your configuration as a struct</li> <li>Pick a format, and include a the required serde features (e.g. <code>yaml</code>)</li> <li>Deserialize your configuration ( <code>serde::from_str</code>)</li> </ol> <p>Most times that is more than enough and makes for a simple, maintainable code that can also be evolved by evolving the shape of your struct.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust"><span class="z-entity z-name z-impl z-rust">Config</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">load</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">path</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>Path</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-storage z-type z-rust">Self</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">serde<span class="z-punctuation z-accessor z-rust">::</span></span>from_str<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-bitwise z-rust">&amp;</span><span class="z-meta z-path z-rust">fs<span class="z-punctuation z-accessor z-rust">::</span></span>read_to_string<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>path</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-keyword z-operator z-rust">?</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-keyword z-operator z-rust">?</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-keyword z-operator z-range z-rust">..</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Some use cases call for “upgrading” configuration loading in two possible ways:</p> <ol> <li>Local vs. global configuration, and their relationships. That is, read the current folder config file, and “bubble up” the folder hierarchy, completing the rest of missing configuration with each new configuration file you find until you get to the user-global configuration file residing at something like <code>~/.your-app/config.yaml</code></li> <li>Layered configuration inputs. That is, reading from a local <code>config.yaml</code> but, if a certain value was provided via an environment flag or CLI flag, override what was found in <code>config.yaml</code> with that value. This calls for a library that can provide alignment of configuration keys for various formats: YAML, CLI flags, environment variables, and so on.</li> </ol> <p>While I strongly urge you to keep things simple and evolve (just use <code>serde</code> loading), I found both of these libraries really robust at reading layered configuration, and do pretty much the same thing:</p> <ul> <li><a href="https://lib.rs/crates/figment">https://lib.rs/crates/figment</a></li> <li><a href="https://github.com/mehcode/config-rs">https://github.com/mehcode/config-rs</a></li> </ul> <h2 id="colors-styling-and-the-terminal">Colors, Styling, and the Terminal</h2> <p>In today’s world, it’s become completely OK to <em>express yourself</em> through styling in the terminal. That means colors, sometimes RGB colors, emoji, animation, and more. Using colors, unicode, risks moving away from being compatible with a “traditional” Unix terminal, but most libraries can detect and downgrade the experience as needed.</p> <h2 id="colors">Colors</h2> <p>If you’re not using any other terminal UX library, <code>owo-colors</code> is great, minimal, resource-friendly, and fun:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-keyword z-other z-rust">use</span> <span class="z-meta z-path z-rust">owo_colors<span class="z-punctuation z-accessor z-rust">::</span></span>OwoColorize<span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">main</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> Foreground colors </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>My number is <span class="z-constant z-other z-placeholder z-rust">{:#x}</span>!<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-separator z-rust">,</span> <span class="z-constant z-numeric z-float z-rust">10.</span><span class="z-support z-function z-rust">green</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> Background colors </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>My number is not <span class="z-constant z-other z-placeholder z-rust">{}</span>!<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-separator z-rust">,</span> <span class="z-constant z-numeric z-float z-rust">4.</span><span class="z-support z-function z-rust">on_red</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>If you’re using something like <code>dialoguer</code> for prompts, it’s worth examining what it uses for colors. In this case, it uses <code>console</code> for terminal manipulation. With <code>console</code> you can style it in this way:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-keyword z-other z-rust">use</span> <span class="z-meta z-path z-rust">console<span class="z-punctuation z-accessor z-rust">::</span></span>Style<span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> cyan <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">Style<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">cyan</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>This is <span class="z-constant z-other z-placeholder z-rust">{}</span> neat<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-separator z-rust">,</span> cyan<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">apply_to</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>quite<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></code></pre> <p>A bit different, but not too different.</p> <h2 id="emoji">Emoji</h2> <p>Thinking about emojis: they’re a form of <a href="https://en.wikipedia.org/wiki/List_of_emoticons">expression</a>. So, 🙂, <code>:-)</code> and <code>[smiling]</code> are all the same expression but different <em>medium</em>. You want the emoji where you’ve got good unicode support, the text smiley on text terminals with no unicode, and the verbose <code>smiling</code> when you want searchable text, or for a more accessible and readable output for visually impaired.</p> <p>One other tip to remember is that Emoji can look different in different terminals. On Windows, you have the old terminal <code>cmd.exe</code> and Powershell, and they are radically different in how they render Emoji from Linux and macOS terminals (while Linux and macOS Emoji rendering are pretty close).</p> <p>For that matter, it’s best to abstract your literal emojis under variables. It can be just a bunch of literals with your own switching logic or something fancier with a <code>fmt::Display</code> implementation.</p> <p>You might want to switch based on a matrix of requirements:</p> <ul> <li>Operating system</li> <li>Terminal feature support (unicode, istty)</li> <li>User requested value (did they ask for no emojis specifically?)</li> </ul> <p>There’s a <a href="https://github.com/console-rs/console/blob/ab9cd9dd7e65c11e050988e9a706a991ed7749cc/src/utils.rs#L691">great implementation of that</a> idea (though not as extensive) in <code>console</code></p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-keyword z-other z-rust">use</span> <span class="z-meta z-path z-rust">console<span class="z-punctuation z-accessor z-rust">::</span></span>Emoji<span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>[3/4] <span class="z-constant z-other z-placeholder z-rust">{}</span>Downloading ...<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-separator z-rust">,</span> Emoji<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>🚚 <span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span><span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>[4/4] <span class="z-constant z-other z-placeholder z-rust">{}</span> Done!<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-separator z-rust">,</span> Emoji<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>✨<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>:-)<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <h2 id="tables">Tables</h2> <p>One of the most flexible libraries for printing and formatting tables is <a href="https://github.com/zhiburt/tabled">tabled</a>. What makes a table printing library flexible?</p> <ul> <li>Supporting “free form data” — just a set of rows and column names</li> <li>Supporting typed records through <code>serde</code> so you just give it a bunch of typed element in a Vec</li> <li>Formatting and shaping: alignment, spacing, spanning, and more</li> <li>Colors support — this one isn’t that easy, for when you’re calculating a table layout, you need to consider ANSI codes which makes a string byte-by-byte longer and hard to predict</li> <li>And much more</li> </ul> <p><code>tabled</code> does all that, and is great. If you’re looking for printing table results, or just table-layout results (such as page-formatting results), don’t look for anything else, this is it.</p> <p><img src="https://rng0.io/blog/building-cli-apps-in-rust-what-you-should-consider/2.png" alt="" /></p> <h2 id="prompts">Prompts</h2> <p><code>dialoguer</code> is the most widely used prompting library out there at the moment, and is rock solid. It has almost all of the different prompts you would expect from a versatile CLI app such as checkbox, option selects, and fuzzy selects.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> items <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-support z-macro z-rust">vec!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>Item 1<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>item 2<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-storage z-type z-rust">let</span> selection <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">FuzzySelect<span class="z-punctuation z-accessor z-rust">::</span></span>with_theme<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-bitwise z-rust">&amp;</span><span class="z-meta z-path z-rust">ColorfulTheme<span class="z-punctuation z-accessor z-rust">::</span></span>default<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span><span class="z-source z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">items</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-bitwise z-rust">&amp;</span>items</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span><span class="z-source z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">default</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-constant z-numeric z-integer z-decimal z-rust">0</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span><span class="z-source z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">interact_on_opt</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-keyword z-operator z-bitwise z-rust">&amp;</span><span class="z-meta z-path z-rust">Term<span class="z-punctuation z-accessor z-rust">::</span></span>stderr<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"> </span><span class="z-source z-rust"><span class="z-keyword z-control z-rust">match</span> selection <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Some</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>index</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> <span class="z-keyword z-operator z-rust">=&gt;</span> <span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>User selected item : <span class="z-constant z-other z-placeholder z-rust">{}</span><span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-separator z-rust">,</span> items<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">[</span>index<span class="z-punctuation z-section z-group z-end z-rust">]</span></span><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">None</span> <span class="z-keyword z-operator z-rust">=&gt;</span> <span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>User did not select anything<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></code></pre> <p>It has one major issue — <a href="https://github.com/console-rs/dialoguer/issues/95">which is a lack of a testability story</a>. That is, if you want to test your code and it depends on it, you have a hard dependency on the terminal (your tests will hang).</p> <p>What you would expect is to have some kind of IO abstraction facility that you can inject on tests, feed it keystrokes programmatically and verify they’ve been read and that the appropriate actions have been taken.</p> <p>Another library is <code>inquire</code> , but it too suffers from not having a testing story, and you can see how complex such a thing might be in an <a href="https://github.com/mikaelmello/inquire/issues/71">issue I'm tracking.</a></p> <p>The good news is that you have a pretty good testing story with the <em>much less popular</em> library <code>requestty</code> though, either way when injecting that IO abstraction layer, you have to also think about mutability and ownership:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-struct z-rust">struct</span> </span><span class="z-meta z-struct z-rust"><span class="z-meta z-generic z-rust"><span class="z-entity z-name z-struct z-rust">Prompt</span><span class="z-meta z-generic z-rust"><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-storage z-modifier z-lifetime z-rust">&#39;a</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span></span><span class="z-meta z-struct z-rust"> </span><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">config</span><span class="z-punctuation z-separator z-type z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-modifier z-lifetime z-rust">&#39;a</span> Config, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">events</span><span class="z-punctuation z-separator z-type z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Option</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust">TestEvents<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust">IntoIter<span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>KeyEvent<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-variable z-other z-member z-rust">show_progress</span><span class="z-punctuation z-separator z-type z-rust">:</span> <span class="z-storage z-type z-rust">bool</span>, </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span><span class="z-source z-rust"> </span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span></span><span class="z-meta z-impl z-rust"><span class="z-meta z-generic z-rust"><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-storage z-modifier z-lifetime z-rust">&#39;a</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-impl z-rust"> <span class="z-entity z-name z-impl z-rust">Prompt</span><span class="z-meta z-generic z-rust"><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-storage z-modifier z-lifetime z-rust">&#39;a</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">build</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">config</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-modifier z-lifetime z-rust">&#39;a</span> Config, <span class="z-variable z-parameter z-rust">show_progress</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-storage z-type z-rust">bool</span>, <span class="z-variable z-parameter z-rust">events</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Option</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-keyword z-operator z-rust">&amp;</span>RunnerEvents<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-storage z-type z-rust">Self</span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> events </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">and_then</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">|</span></span></span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-variable z-parameter z-rust">evs</span><span class="z-punctuation z-section z-parameters z-end z-rust">|</span></span> </span><span class="z-meta z-function z-closure z-rust">evs<span class="z-punctuation z-accessor z-dot z-rust">.</span>prompt_events<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">as_ref</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">map_or_else</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">|</span></span></span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">|</span></span> </span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-path z-rust">Prompt<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>config<span class="z-punctuation z-separator z-rust">,</span> show_progress</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"> <span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">|</span></span></span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-variable z-parameter z-rust">evs</span><span class="z-punctuation z-section z-parameters z-end z-rust">|</span></span> </span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-path z-rust">Prompt<span class="z-punctuation z-accessor z-rust">::</span></span>with_events<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>config<span class="z-punctuation z-separator z-rust">,</span> evs<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">clone</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-closure z-rust"> </span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-keyword z-operator z-range z-rust">...</span> </span></span></span></code></pre> <p>It isn’t a pretty sight but it makes for a well tested CLI interaction flow.</p> <p><strong>What other options do you have to handle testing?</strong></p> <ul> <li>Trust the rock-stable state of <em>dialoguer</em> and just not test the interaction parts of your app</li> <li>Test the interaction via black-box testing (more on testing later). You can get pretty far with this approach in terms of your ROI on testing</li> <li>Build your own test rig, with a switchable UI interaction layer, where you completely replace it in test time with something that replays action (again, your own custom code). This does mean that the real code dealing with prompts and selects will never be tested, as little code as that may be</li> </ul> <h2 id="status-and-progress">Status and Progress</h2> <p><code>indicatif</code> is the gold standard Rust library for status and progress bars. There’s only one great library and that’s a good thing because we’re not stuck in the paradox of options, so use this one!</p> <p><img src="https://rng0.io/blog/building-cli-apps-in-rust-what-you-should-consider/3.png" alt="" /></p> <h2 id="operability">Operability</h2> <p>There are mostly two flavors of logging for operability in Rust and you can use both or one of those:</p> <ul> <li><strong>Logging</strong> — and what evolved as a standard: <a href="https://lib.rs/crates/log">https://lib.rs/crates/log</a> where people mostly combine it with <a href="https://lib.rs/crates/env_logger">env_logger</a> which is simple and really easy to use.</li> </ul> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-path z-rust">env_logger<span class="z-punctuation z-accessor z-rust">::</span></span>init<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-support z-macro z-rust">info!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>starting up<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> RUST_LOG=INFO ./main<span class="z-keyword z-control z-regexp z-set z-begin z-shell">[</span>2018<span class="z-keyword z-operator z-word z-shell">-</span>11<span class="z-keyword z-operator z-word z-shell">-</span>03T06:09:06Z INFO default<span class="z-keyword z-control z-regexp z-set z-end z-shell">]</span> starting up</span> </span></code></pre> <ul> <li><strong>Tracing</strong> — for this, you need to use the one <code>tracing</code> crate and the ecosystem that Rust has (luckily there’s just one!). You can start with <code>tracing-tree</code> but you can also later decide to plug in telemetry and third-party SDKs as well as print flame graphs as you would expect from a tracing infrastructure</li> </ul> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">server{host=&quot;localhost&quot;, port=8080} </span><span class="z-text z-plain"> 0ms INFO starting </span><span class="z-text z-plain"> 300ms INFO listening </span><span class="z-text z-plain"> conn{peer_addr=&quot;82.9.9.9&quot;, port=42381} </span><span class="z-text z-plain"> 0ms DEBUG connected </span><span class="z-text z-plain"> 300ms DEBUG message received, length=2 </span><span class="z-text z-plain"> conn{peer_addr=&quot;8.8.8.8&quot;, port=18230} </span><span class="z-text z-plain"> 300ms DEBUG connected </span><span class="z-text z-plain"> conn{peer_addr=&quot;82.9.9.9&quot;, port=42381} </span><span class="z-text z-plain"> 600ms WARN weak encryption requested, algo=&quot;xor&quot; </span><span class="z-text z-plain"> 901ms DEBUG response sent, length=8 </span><span class="z-text z-plain"> 901ms DEBUG disconnected </span><span class="z-text z-plain"> conn{peer_addr=&quot;8.8.8.8&quot;, port=18230} </span><span class="z-text z-plain"> 600ms DEBUG message received, length=5 </span><span class="z-text z-plain"> 901ms DEBUG response sent, length=8 </span><span class="z-text z-plain"> 901ms DEBUG disconnected </span><span class="z-text z-plain"> 1502ms WARN internal error </span><span class="z-text z-plain"> 1502ms INFO exit </span></code></pre> <p>Tracing lets you instrument your code very easily by decorating functions:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">tracing</span>::<span class="z-variable z-annotation z-rust">instrument</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">level <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>trace<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> skip_all<span class="z-punctuation z-separator z-rust">,</span> err</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">is_archive</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">file</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>File, <span class="z-variable z-parameter z-rust">fval</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span>[String]</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Option</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-storage z-type z-rust">bool</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span> </span></span></span></code></pre> <p>And you can have the option for capturing arguments, return values and errors from the function automatically.</p> <h2 id="error-handling">Error Handling</h2> <p>This is a big one in Rust. Simple because errors went through a big stage of evolution. There have been a few libraries that took off, then died out, and then a few more that took off and died out.</p> <p>All in all, it was a fantastic process. Between each cycle the Rust ecosystem had real learnings done, and improvements made, and today we have a few libraries that are really great.</p> <p>The down side of it, is that depending on the code you’ll be reading, examples here and there, and open source projects — you’ll need to make a mental note of what libraries it uses and to which era of error libraries it belongs.</p> <p>So, as of writing this article, these are the libraries that I found to be perfect for a CLI app. Here too, I divide my thinking into “app errors” and “library errors” where for library errors you want to use error types that rely on the standard library and not force your users to use a specialized error library.</p> <p><img src="https://rng0.io/blog/building-cli-apps-in-rust-what-you-should-consider/4.png" alt="" /></p> <ul> <li>App errors: <code>eyre</code> which is a close relative of <code>anyhow</code>, but has a really great error-reporting story with libraries such as <code>color-eyre</code></li> <li>Library errors: I used to use <code>thiserror</code> but then moved on to <code>snafu</code> which I use for everything. <code>snafu</code> gives you all the advantages of <code>this-error</code> but with the ergonomics of <code>anyhow</code> or <code>eyre</code></li> </ul> <p>And then, I use libraries that improve their error reporting. Majorly I use <code>fs_err</code> instead of <code>std::fs</code> which has the same API but more elaborate and human-friendly errors, for example:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">failed to open file `does not exist.txt` </span><span class="z-text z-plain"> caused by: The system cannot find the file specified. (os error 2) </span></code></pre> <p>Instead of</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">The system cannot find the file specified. (os error 2) </span></code></pre> <h2 id="testing">Testing</h2> <p>I find <strong>balancing</strong> test types and strategies in Rust can be a very delicate but rewarding task. That is, Rust is safe. It doesn’t mean your code works well out of the gate, but it does mean that on top of being a statically typed language with types that guard against many programming mistakes, it’s also safe in the sense that it removes a wide array of programmer errors relating to sharing data and ownership.</p> <p>I would say carefully, that my Rust code has fewer tests compared to my Ruby or JavaScript code, and is more sound.</p> <p>I find that this property of Rust, also brings back blackbox testing in a big way. Because once you’ve tested some of the insides, combining modules and integrating them is rather safe by virtue of the compiler.</p> <p>So all in all my testing strategy for Rust CLI apps is:</p> <ol> <li><strong>Unit tests</strong> — testing logic in functions and modules</li> <li><strong>Integration tests</strong> — per need basis, between modules and testing out complex interaction workflows</li> <li><strong>Blackbox tests</strong> — using tools like <code>try_cmd</code> to run a CLI session, provide input, capture output, and snapshot the resulting state to be approved and saved</li> </ol> <p>I use snapshot testing where I can because there’s no point in left-right-coding in tests:</p> <ul> <li><code>insta</code> - <a href="https://docs.rs/insta/latest/insta/">https://docs.rs/insta/latest/insta/</a>, takes care of dev workflow, snapshotting, review, and extras such as redaction and various serialization formats for the snapshots</li> </ul> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">test</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">test_simple</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-path z-rust">insta<span class="z-punctuation z-accessor z-rust">::</span></span>assert_yaml_snapshot<span class="z-keyword z-operator z-logical z-rust">!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-function z-rust">calculate_value</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <ul> <li><code>trycmd</code> - <a href="https://lib.rs/crates/trycmd">https://lib.rs/crates/trycmd</a>, is really robust and works well, and is fantastically simple. You write your tests as a markdown file and it will parse, run the commands that are embedded and keep track of what results should be against the result in that same markdown file, so your tests are living documentation as well</li> </ul> <pre data-lang="markdown" class="language-markdown z-code"><code class="language-markdown" data-lang="markdown"><span class="z-text z-html z-markdown"><span class="z-meta z-paragraph z-markdown"><span class="z-meta z-code-fence z-definition z-begin z-text z-markdown-gfm"><span class="z-punctuation z-definition z-raw z-code-fence z-begin z-markdown">```</span><span class="z-constant z-other z-language-name z-markdown">console</span> </span><span class="z-markup z-raw z-code-fence z-markdown-gfm"></span></span></span><span class="z-text z-html z-markdown"><span class="z-meta z-paragraph z-markdown"><span class="z-markup z-raw z-code-fence z-markdown-gfm">$ my-cmd </span></span></span><span class="z-text z-html z-markdown"><span class="z-meta z-paragraph z-markdown"><span class="z-markup z-raw z-code-fence z-markdown-gfm">Hello world </span></span></span></code></pre> <h2 id="releasing-and-shipping-a-binary">Releasing and Shipping a Binary</h2> <p>Over time, I’ve shaped my release workflow <strong>into starter projects and tools</strong>, so instead of lining out and showcasing how to build a workflow from scratch, just use those tools and projects. And if you’re curious — read their code.</p> <p>If you want to have the easiest time here you can check these out:</p> <ul> <li><a href="https://github.com/rusty-ferris-club/rustwrap">rustwrap</a> — for releasing built binaries from Github releases into homebrew or npm</li> <li><a href="https://github.com/jondot/xtaskops">xtaskops</a> — for moving some logic from your CI into Rust in the form of the xtask pattern — read more in <a href="https://rng0.io/blog/running-rust-tasks-with-xtask-and-xtaskops/">“Running Rust Tasks with xtask and xtaskops”</a></li> <li><a href="https://www.notion.so/The-Ultimate-Guide-to-Building-CLI-Apps-in-Rust-d2c4573be3bf4d6db299d52e358e5b72">rust-starter</a> — for taking the ready-made CI workflows for streamlined testing, lint, and release</li> </ul> <p>If you have a proper Rust binary, you should also consider releasing to cargo bins, look at <code>cargo-binstall</code> for more on that.</p> Compile Time Feature Flags in Rust: Why, How, and When? Wed, 14 Dec 2022 00:00:00 +0000 Unknown https://rng0.io/blog/compile-time-feature-flags-in-rust-why-how-and-when/ https://rng0.io/blog/compile-time-feature-flags-in-rust-why-how-and-when/ <h3 id="and-what-s-the-tradeoff">And what’s the tradeoff?</h3> <p><img src="https://rng0.io/blog/compile-time-feature-flags-in-rust-why-how-and-when/1.png" alt="" /></p> <p>The ability to pick compile time features in Rust can improve your code's performance, size, maintainability, safety, and portability.</p> <p>Below are a few arguments for why you should proactively use features when consuming dependencies and offer those to other library users.</p> <h2 id="performance">Performance</h2> <p>Using feature flags in Rust can improve the performance of the resulting code. By only including the code that is needed for a specific application, you can avoid the overhead of unused or unnecessary code.</p> <p>Though there are compiler optimizations to remove dead code, still, this can result in faster and more efficient programs (and make the compiler’s life easier).</p> <h2 id="size">Size</h2> <p>The overall size of the resulting binary is influenced by which dependencies you include and how you use them.</p> <p>Feature selection can help the resulting binary be smaller, benefiting applications that need to be distributed or deployed to resource-constrained environments.</p> <h2 id="maintainability">Maintainability</h2> <p>I’ve recently had a breaking upstream dependency, where I was lucky enough to have that upstream breaking code under a feature flag — for a feature I was not using.</p> <p>While waiting for the upstream library to update, I removed the feature for my local project, which was building fine again. This means you can improve Rust code's maintainability by allowing developers to include or exclude specific functionality selectively.</p> <h2 id="security">Security</h2> <p>Statistically speaking — the more code you depend on, the higher the chance of a security issue. Depending on only the features you need to lower the odds of a security issue is security-by-design thinking, and a crate that offers itself “in chunks” is helping that happen.</p> <p>There are also ways to select different implementations of the same functionality based on how comfortable you are with the safety of an implementation. For example, you might prefer a Rust-native TLS implementation over a C-based one because Rust is a safe language, and some <a href="https://github.com/seanmonstar/reqwest/blob/master/Cargo.toml#L29-L44">crates like Reqwest</a> offer a selection of TLS backends.</p> <h2 id="portability">Portability</h2> <p>As a compiled language, an important aspect of feature flags is improving your code's portability.</p> <p>You can selectively include or exclude specific functionality to make your code more portable across different platforms and environments.</p> <h2 id="how-does-c-c-compare">How Does C/C++ Compare?</h2> <p>C and C++ have historically been the archetypes of compiled portable code deployed to many platforms and CPU architectures.</p> <p>C++ does not have a built-in feature directly equivalent to the ability to pick compile time features in Rust. However, <a href="https://stackoverflow.com/questions/4469645/dos-and-donts-of-conditional-compile">C++ does have a number of preprocessor directives</a> that can be used to include or exclude certain code at compile time selectively.</p> <p>This can provide some of the same benefits as feature flags in Rust. Still, it’s messy and hard to discover — both as a programmer looking to build into an existing codebase and as a consumer looking to enable or disable features.</p> <h2 id="feature-flags-the-building-blocks">Feature Flags: The Building Blocks</h2> <p><img src="https://rng0.io/blog/compile-time-feature-flags-in-rust-why-how-and-when/2.png" alt="" /></p> <p>To enable a specific feature flag for a specific crate, you can use the <code>default-features = false</code> and <code>features</code> attributes in the crate's <code>Cargo.toml</code> file.</p> <p>For example:</p> <pre data-lang="toml" class="language-toml z-code"><code class="language-toml" data-lang="toml"><span class="z-source z-toml"><span class="z-punctuation z-definition z-table z-begin z-toml">[</span><span class="z-meta z-tag z-table z-toml"><span class="z-entity z-name z-table z-toml">dependencies</span></span><span class="z-punctuation z-definition z-table z-end z-toml">]</span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">my-crate</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-punctuation z-definition z-inline-table z-begin z-toml">{</span> <span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">default-features</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-constant z-language z-toml">false</span><span class="z-punctuation z-separator z-inline-table z-toml">,</span> <span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">features</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-punctuation z-definition z-array z-begin z-toml">[</span><span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>my-feature<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span><span class="z-punctuation z-definition z-array z-end z-toml">]</span> <span class="z-punctuation z-definition z-inline-table z-end z-toml">}</span> </span></code></pre> <p>To enable a feature flag for a specific piece of code, you can use the <code>#[cfg(feature = "my-feature")]</code> attribute. For example:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>my-feature<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">my_function</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> Code that is only included when the &quot;my-feature&quot; flag is enabled </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>To enable a feature flag for a specific module, you can use the <code>#[cfg(feature = "my-feature")]</code> attribute on the <code>mod</code> declaration. For example:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>my-feature<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-storage z-type z-module z-rust">mod</span> <span class="z-entity z-name z-module z-rust">my_module</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> Code that is only included when the &quot;my-feature&quot; flag is enabled </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>To enable a feature flag for a specific struct or enum with <code>derive</code>, you can use the <code>#[cfg_attr(feature = "my-feature", derive(...))]</code> attribute. For example:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg_attr</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>my-feature<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> <span class="z-meta z-function-call z-rust"><span class="z-variable z-function z-rust">derive</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-function-call z-rust">Debug<span class="z-punctuation z-separator z-rust">,</span> PartialEq</span><span class="z-meta z-function-call z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-storage z-type z-struct z-rust">struct</span> </span><span class="z-meta z-struct z-rust"><span class="z-entity z-name z-struct z-rust">MyStruct</span> </span><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> Fields and methods that are only included when the &quot;my-feature&quot; flag is enabled </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-struct z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Here’s how to enable or disable support for a specific platform:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">target_os <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>linux<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-storage z-type z-module z-rust">mod</span> <span class="z-entity z-name z-module z-rust">linux_specific_code</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> Linux-specific code goes here... </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>And how to enable or disable a specific implementation of a trait:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>special_case<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-storage z-type z-impl z-rust">impl</span> </span><span class="z-meta z-impl z-rust">MyTrait <span class="z-keyword z-other z-rust">for</span></span><span class="z-meta z-impl z-rust"> <span class="z-entity z-name z-impl z-rust">MyType</span> </span><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> Implementation of trait for special case goes here... </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-impl z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>How to enable or disable a specific test case:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>expensive_tests<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">test</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">test_expensive_computation</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> Test that performs expensive computation goes here... </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Here’s the code to enable or disable a specific benchmark:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>long_benchmarks<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">bench</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">bench_long_running_operation</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">b</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-modifier z-rust">mut</span> Bencher</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> Benchmark for a long-running operation goes here... </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>To enable a feature only when multiple flags are set, you can use the <code>#[cfg(all(feature1, feature2, ...))]</code> attribute. For example, to enable a <code>my_function()</code> only when both the <code>my_feature1</code> and <code>my_feature2</code> flags are set:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function-call z-rust"><span class="z-variable z-function z-rust">all</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-function-call z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>my_feature1<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>my_feature2<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-function-call z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">my_function</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> code for my_function </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>To enable a feature only when one of the multiple flags is set, you can use the <code>#[cfg(any(feature1, feature2, ...))]</code> attribute. For example, to enable a <code>my_function()</code> when either the <code>my_feature1</code> or <code>my_feature2</code> flag is set:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function-call z-rust"><span class="z-variable z-function z-rust">any</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-function-call z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>my_feature1<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>my_feature2<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-function-call z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">my_function</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> code for my_function </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h2 id="feature-flags-illustrated">Feature Flags Illustrated</h2> <p><img src="https://rng0.io/blog/compile-time-feature-flags-in-rust-why-how-and-when/3.png" alt="" /></p> <p>Same module, but point to a different path for implementation, then pull out a function to expose from that module with <code>pub use</code>.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-comment z-line z-documentation z-rust"><span class="z-punctuation z-definition z-comment z-rust">//!</span> Signal monitor </span></span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">unix</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">path</span> <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>unix.rs<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-storage z-type z-module z-rust">mod</span> <span class="z-entity z-name z-module z-rust">imp</span><span class="z-punctuation z-terminator z-rust">;</span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">windows</span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">path</span> <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>windows.rs<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-storage z-type z-module z-rust">mod</span> <span class="z-entity z-name z-module z-rust">imp</span><span class="z-punctuation z-terminator z-rust">;</span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function-call z-rust"><span class="z-variable z-function z-rust">not</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-function-call z-rust"><span class="z-meta z-function-call z-rust"><span class="z-variable z-function z-rust">any</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-function-call z-rust">windows<span class="z-punctuation z-separator z-rust">,</span> unix</span><span class="z-meta z-function-call z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span></span><span class="z-meta z-function-call z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">path</span> <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>other.rs<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-storage z-type z-module z-rust">mod</span> <span class="z-entity z-name z-module z-rust">imp</span><span class="z-punctuation z-terminator z-rust">;</span></span> </span><span class="z-source z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-keyword z-other z-rust">use</span> <span class="z-meta z-path z-rust"><span class="z-variable z-language z-rust">self</span><span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">imp<span class="z-punctuation z-accessor z-rust">::</span></span>create_signal_monitor<span class="z-punctuation z-terminator z-rust">;</span> </span></code></pre> <p>See <code> https://github.com/shadowsocks/shadowsocks-rust/blob/master/src/monitor/mod.rs</code></p> <p>When different components have the same implementation: you can offer everything under the sun without any disadvantage because only the features selected get compiled in.</p> <p>The tradeoff is that now you have a bigger test matrix, which grows combinatorially with every new alternative.</p> <p>In this example, the library lets you pick any allocator you can think of because allocators have a well-defined interface and require no work on your part to swap:</p> <p><img src="https://rng0.io/blog/compile-time-feature-flags-in-rust-why-how-and-when/4.png" alt="" /></p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-comment z-line z-documentation z-rust"><span class="z-punctuation z-definition z-comment z-rust">//!</span> Memory allocator </span></span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>jemalloc<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">global_allocator</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-storage z-type z-rust">static</span> <span class="z-constant z-other z-rust">ALLOC</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-path z-rust">jemallocator<span class="z-punctuation z-accessor z-rust">::</span></span>Jemalloc <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">jemallocator<span class="z-punctuation z-accessor z-rust">::</span></span>Jemalloc<span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>tcmalloc<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">global_allocator</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-storage z-type z-rust">static</span> <span class="z-constant z-other z-rust">ALLOC</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-path z-rust">tcmalloc<span class="z-punctuation z-accessor z-rust">::</span></span>TCMalloc <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">tcmalloc<span class="z-punctuation z-accessor z-rust">::</span></span>TCMalloc<span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>mimalloc<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">global_allocator</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-storage z-type z-rust">static</span> <span class="z-constant z-other z-rust">ALLOC</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-path z-rust">mimalloc<span class="z-punctuation z-accessor z-rust">::</span></span>MiMalloc <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">mimalloc<span class="z-punctuation z-accessor z-rust">::</span></span>MiMalloc<span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>snmalloc<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">global_allocator</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-storage z-type z-rust">static</span> <span class="z-constant z-other z-rust">ALLOC</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-path z-rust">snmalloc_rs<span class="z-punctuation z-accessor z-rust">::</span></span>SnMalloc <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">snmalloc_rs<span class="z-punctuation z-accessor z-rust">::</span></span>SnMalloc<span class="z-punctuation z-terminator z-rust">;</span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>rpmalloc<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">global_allocator</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-storage z-type z-rust">static</span> <span class="z-constant z-other z-rust">ALLOC</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-path z-rust">rpmalloc<span class="z-punctuation z-accessor z-rust">::</span></span>RpMalloc <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">rpmalloc<span class="z-punctuation z-accessor z-rust">::</span></span>RpMalloc </span></code></pre> <p>In this example, you see how to let your users “layer in” the functionality they need, where you can pick how much deeper you want to go:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-comment z-line z-documentation z-rust"><span class="z-punctuation z-definition z-comment z-rust">//!</span> Service launchers </span></span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-module z-rust">mod</span> <span class="z-entity z-name z-module z-rust">genkey</span><span class="z-punctuation z-terminator z-rust">;</span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>local<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-module z-rust">mod</span> <span class="z-entity z-name z-module z-rust">local</span><span class="z-punctuation z-terminator z-rust">;</span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>manager<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-module z-rust">mod</span> <span class="z-entity z-name z-module z-rust">manager</span><span class="z-punctuation z-terminator z-rust">;</span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>server<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-module z-rust"><span class="z-storage z-modifier z-rust">pub</span> <span class="z-storage z-type z-module z-rust">mod</span> <span class="z-entity z-name z-module z-rust">server</span><span class="z-punctuation z-terminator z-rust">;</span></span> </span></code></pre> <p>In the below example, you can use blocks to “artificially” scope in entire pieces of code under a feature:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust">feature <span class="z-keyword z-operator z-rust">=</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>local-tunnel<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"> app <span class="z-keyword z-operator z-assignment z-rust">=</span> app<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">arg</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-meta z-path z-rust">Arg<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>FORWARD_ADDR<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">short</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-single z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&#39;</span>f<span class="z-punctuation z-definition z-string z-end z-rust">&#39;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">long</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>forward-addr<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">num_args</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-constant z-numeric z-integer z-decimal z-rust">1</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">action</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">ArgAction<span class="z-punctuation z-accessor z-rust">::</span></span>Set</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">requires</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>LOCAL_ADDR<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">value_parser</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">vparser<span class="z-punctuation z-accessor z-rust">::</span></span>parse_address</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">required_if_eq</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>PROTOCOL<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>tunnel<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">help</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>Forwarding data directly to this address (for tunnel)<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> </span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></code></pre> <p>In this example, we inline empty implementations because why pay the price of a function call if its body always returns a simplistic and empty-ish value? ( <code>Ok(()</code>).</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">cfg</span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function-call z-rust"><span class="z-variable z-function z-rust">all</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-function-call z-rust"><span class="z-meta z-function-call z-rust"><span class="z-variable z-function z-rust">not</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-function-call z-rust">windows</span><span class="z-meta z-function-call z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-separator z-rust">,</span> <span class="z-meta z-function-call z-rust"><span class="z-variable z-function z-rust">not</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span></span><span class="z-meta z-function-call z-rust">unix</span><span class="z-meta z-function-call z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span></span><span class="z-meta z-function-call z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span></span></span><span class="z-meta z-annotation z-parameters z-rust"><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">inline</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">set_common_sockopt_after_connect_sys</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-keyword z-operator z-rust">_</span>: <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-meta z-path z-rust">tokio<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">net<span class="z-punctuation z-accessor z-rust">::</span></span>TcpStream, <span class="z-keyword z-operator z-rust">_</span>: <span class="z-keyword z-operator z-rust">&amp;</span>ConnectOpts</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-path z-rust">io<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-punctuation z-section z-group z-end z-rust">)</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h2 id="last-but-not-least-what-s-the-tradeoff">Last but Not Least: What’s the Tradeoff?</h2> <p>If features are so powerful and shed away a lot of C/C++’s primitive ways of doing conditional code compilation, why not use it everywhere and always? Here are a few things you should consider.</p> <ul> <li><strong>Using too many features is a real thing.</strong> In the imaginary and extreme case, imagine you had a feature on every module and function. That would require your consumers to solve a very hard puzzle of understanding how to compose your library from its discrete features. This is the danger of features. You want to be modest with the number of features you’re offering to reduce a cognitive load and for those features to be things people care about removing or adding.</li> <li><strong>Testing is another big deal with features.</strong> You never know which combination of features your users will select, and every combination selects a different set of code — and those code pieces have to interoperate smoothly both in the compilation (successfully compile). In logic (not introduce bugs), you need to test a combination of all features with every other feature and create a powerset of features!</li> <li><strong>You can automate that with</strong> <code>xtaskops::powerset</code>— see more here: <a href="https://github.com/jondot/xtaskops">https://github.com/jondot/xtaskops</a>.</li> </ul> Introducing Hyperstack — The Pragmatic Node.js Web Framework for Builders Tue, 29 Nov 2022 00:00:00 +0000 Unknown https://rng0.io/blog/introducing-hyperstack-the-pragmatic-node-js-web-framework-for-builders/ https://rng0.io/blog/introducing-hyperstack-the-pragmatic-node-js-web-framework-for-builders/ <h3 id="a-look-at-our-latest-open-source-tool-for-programmers">A look at our latest open-source tool for programmers</h3> <p><img src="https://rng0.io/blog/introducing-hyperstack-the-pragmatic-node-js-web-framework-for-builders/1.png" alt="Complex architecture" /></p> <p><a href="https://github.com/hyperstackjs/hyperstack">Hyperstack</a> is a new open-source Node.js web framework for builders. It’s not reinventing the wheel, it’s just wrapping a bunch of great wheels and good parts for you to use — in one nice Rails-like package.</p> <p>Good software is simple software that solves complex problems. Building simple software usually requires following some flavor of principles, design patterns and strong opinions that help us navigate the perils of <a href="https://en.wikipedia.org/wiki/The_Paradox_of_Choice">the paradox of choice</a>.</p> <p>When you don’t have these principles and best practices to guide simplicity, I’ve seen engineers kick the tires for fun with the year’s trending microservice API framework, and before you know it they’re legs deep in building a microservice architecture for a painfully simple web app. “ <em>That’s how we’re supposed to do it, that’s how to build scalable software,”</em> they thought.</p> <p>For many, it’s becoming easy to build complex software, and hard to start something simple.</p> <h2 id="solving-simple-problems-with-simple-software">Solving simple problems with simple software</h2> <p><img src="https://rng0.io/blog/introducing-hyperstack-the-pragmatic-node-js-web-framework-for-builders/2.png" alt="Ice cream with servers in it" /></p> <p>A single app, with a single relational database, has been perfect for decades for building great software and still is great for starting out on the right foot to create simple software for projects that are fun to build.</p> <p><a href="https://rubyonrails.org/">Ruby on Rails</a> enabled exactly that. You build a <a href="https://m.signalvnoise.com/the-majestic-monolith/"><em>majestic monolith</em></a> that solves simple <em>and</em> complex problems until you can’t anymore.</p> <p>O <em>nly then</em>, when you’re smarter about everything, when you have all of the use cases sorted out with piles of data and experience — <em>then</em> you start splitting and building microservices.</p> <p>This approach worked for: GitHub, Airbnb, Twitter. All of them started as a monolith. Some of them still are a majestic monolith: <a href="https://gitlab.com/gitlab-org/gitlab">Gitlab</a>. Most of them still use Ruby on Rails.</p> <h2 id="typescript-vs-ruby-in-a-rails-world">TypeScript vs. Ruby in a Rails World</h2> <p><img src="https://rng0.io/blog/introducing-hyperstack-the-pragmatic-node-js-web-framework-for-builders/3.png" alt="Typescript vs Ruby" /></p> <p>Rails is arguably the best web framework that was ever created, and Ruby is the programming language that you have to use to build Rails apps. It most likely influenced everything you’re touching today in software on the Web whether you know it or not.</p> <p>While you still can use Rails, you might prefer using Typescript and Node.js and not Ruby.</p> <p>There are plenty of reasons to prefer Typescript and Node.js for backend development instead of Ruby:</p> <ul> <li><strong>Prior experience for individuals or team</strong>, or prior codebase you’re trying to modernize</li> <li><strong>Code sharing</strong> between server and client is a real thing</li> <li><strong>Infrastructure sharing</strong> with frontend tooling (there are so many!) is a big deal: dev tooling, CI, code quality, leading to whole-team effectiveness over time</li> <li>Lastly, Ruby is a great language, but <strong>not as popular as typescript anymore</strong></li> </ul> <p><img src="https://rng0.io/blog/introducing-hyperstack-the-pragmatic-node-js-web-framework-for-builders/4.png" alt="" /></p> <p>Image: Wikipedia</p> <h2 id="what-does-it-take-to-create-a-rails-in-a-typescript-world"><strong>What does it take to create a Rails in a TypeScript world?</strong></h2> <ul> <li><strong>Models</strong> — a strong ActiveRecord abstraction, that magically knows to connect and operate</li> <li><strong>Database handling</strong>: migration, seeding — and support for multiple databases to allow for better developer experience</li> <li><strong>Controllers</strong>, routing, and authentication and security best practices — to allow for easy API building as well as avoiding <a href="https://www.zdnet.com/home-and-office/networking/github-suspends-member-over-mass-assignment-hack/">common security gotchas</a></li> <li><strong>Mailers</strong> — while you can do emails via 3rd party SaaS, it’s much more productive to build, test, and experiment with emails locally. you need support for that.</li> <li><strong>Background jobs</strong> — because using a raw queue service such as SQS isn’t as ergonomic, or test friendly, as using a dedicated background job framework.</li> <li><strong>Tasks</strong> — you need to fix records, clean up data, or send ad-hoc emails to users. Build a custom task that is fully testable with access to your live app and invoke it just like you use <code>make</code></li> <li><strong>Interactive REPL</strong> — easily access live objects and information about an app remotely or locally</li> <li><strong>Code generation and scaffolding</strong> — build faster by generating models, controllers and more</li> <li><strong>Testing as first-class citizen</strong> — every piece of the framework lends itself to clean, easy testing</li> <li><strong>Configuration and operability</strong> — concerns such as logging, monitoring and app control plane</li> <li><strong>Conventions and opinions</strong> — folder structure, naming, linting, formatting, are all taken off your back so you can focus on building products</li> <li><strong>KISS</strong> — when facing a choice, always pick a simpler path. For example, ActiveRecord over Data Mapper, or any other entity mapping magic. ActiveRecord lends to simpler code.</li> <li><strong>Views</strong> — an automatic way to cut the boilerplate of formatting, serializing responses and error states from the app toward clients</li> </ul> <p><img src="https://rng0.io/blog/introducing-hyperstack-the-pragmatic-node-js-web-framework-for-builders/5.png" alt="" /></p> <p>Here are some of the trials and tribulations if you were to try to assemble such a thing from scratch, specifically for Javascript and Node.js. Surely some of it sounds familiar:</p> <ul> <li>Building this with express you <strong>need to</strong> <a href="https://stackoverflow.com/questions/61790224/node-js-middleware-wrap"><strong>remember about async failures</strong></a>. Yes, <a href="https://koajs.com/">Koa</a> and others are great, but Express works for the vast majority of API use cases, it’s also simple and widely adopted, and is what people usually expect to find in a general-purpose API project.</li> <li><strong>Picking and wiring up loggers</strong> takes can be a mess handling request scoping, process scoping and more, you have to thread the needle through global and local states in an invented world.</li> <li><strong>Building a background jobs infrastructure from scratch will set you up for failure</strong>. There’s way too much to know here to build this from scratch or even pick the right queue system and build the right abstraction and infrastructure on top of it.</li> <li><strong>For testing</strong>, you need to spawn an Express app, remember to clean up databases, set up a schema, load data, and start your tests. All these are infrastructure you have to build and trial and error until it works well for you.</li> </ul> <p>And that’s just the tip of the iceberg. Luckily, I’ve built <a href="https://hyperstackjs.io/">Hyperstack</a> so you don’t have to build this yourself.</p> <h2 id="a-hyperstack-app-in-5-minutes">A Hyperstack App in 5 Minutes</h2> <p><img src="https://rng0.io/blog/introducing-hyperstack-the-pragmatic-node-js-web-framework-for-builders/6.png" alt="Hyperstack" /></p> <p><a href="https://hyperstackjs.io/">Hyperstack</a> is a TypeScript web framework, that has Rails as its north star. It follows Rails closely where it benefits developer experience, but deviates radically where it benefits the pragmatism of building a Node.js app in Typescript.</p> <p>Let’s create our first app:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> pnpm create hyperstack</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cd my-hyper-app</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> pnpm install</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> bin/hyperstack start</span> </span></code></pre> <p><strong>And we’re done!</strong></p> <p><img src="https://rng0.io/blog/introducing-hyperstack-the-pragmatic-node-js-web-framework-for-builders/7.png" alt="Hyperstack running in the terminal" /></p> <p>Hyperstack has created the database, migrated, set up your working development environment, transpiled JavaScript, and configured a fancy API service for you with great defaults such as secure headers, static serving, email previews, and background workers that are in process but with configuration become a fully scalable queue-based background job processors.</p> <p>Kill the process <code>CTRL-C</code> and let’s add our blog <code>article</code>:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> bin/hyperstack g scaffold article title:string body:text</span> </span></code></pre> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">Loaded templates: gen/dist/templates </span><span class="z-text z-plain"> added: src/test/requests/articles/sanity.spec.ts </span><span class="z-text z-plain"> added: src/app/controllers/articles.ts </span><span class="z-text z-plain"> added: src/config/db/migrate/1337_article.js </span><span class="z-text z-plain"> added: src/test/models/article.spec.ts </span><span class="z-text z-plain"> added: src/app/models/article.ts </span><span class="z-text z-plain"> inject: src/app/controllers/index.ts </span><span class="z-text z-plain"> inject: src/app/controllers/index.ts </span><span class="z-text z-plain"> inject: src/app/models/index.ts </span><span class="z-text z-plain">✔ shell: running eslint </span><span class="z-text z-plain"> shell: yarn eslint --fix src/app/models/*.ts src/app/controllers/*.ts src/test/models/*.spec.ts src/test/requests/articles/*.spec.ts src/config/db/migrate/*.js </span></code></pre> <p>You got a controller, model, migration, and automatically written tests for free.</p> <p>Run your app again and let’s start making requests:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> bin/hyperstack start</span> </span></code></pre> <p>I’m using <code>xh</code> but you can use <code>httpie</code> which is similar, or anything you like:</p> <p><strong>List:</strong></p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> xh<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>j</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>b</span> localhost:5150/articles</span> </span><span class="z-source z-shell z-bash"><span class="z-punctuation z-definition z-compound z-braces z-begin z-shell">{</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell"><span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>articles<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span>:</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-keyword z-control z-regexp z-set z-begin z-shell">[</span><span class="z-keyword z-control z-regexp z-set z-end z-shell">]</span></span> </span><span class="z-source z-shell z-bash"><span class="z-punctuation z-definition z-compound z-braces z-end z-shell">}</span> </span></code></pre> <p>It’s empty, so let’s create one article:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> xh<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>j</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>b</span> localhost:5150/articles title=<span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>hello hyperstack<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span> body=<span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>EVH rocks<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span></span> </span><span class="z-source z-shell z-bash"><span class="z-punctuation z-definition z-compound z-braces z-begin z-shell">{</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell"><span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>article<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span>:</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-meta z-group z-expansion z-brace z-shell"><span class="z-punctuation z-section z-expansion z-brace z-begin z-shell">{</span> </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-meta z-group z-expansion z-brace z-shell"> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>title<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span>: <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>hello hyperstack<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span><span class="z-punctuation z-separator z-shell">,</span> </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-meta z-group z-expansion z-brace z-shell"> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>body<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span>: <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>EVH rocks<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span> </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-meta z-group z-expansion z-brace z-shell"> <span class="z-punctuation z-section z-expansion z-brace z-end z-shell">}</span></span></span> </span><span class="z-source z-shell z-bash"><span class="z-punctuation z-definition z-compound z-braces z-end z-shell">}</span> </span></code></pre> <p>Take a look at the list again:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> xh<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>j</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>b</span> localhost:5150/articles</span> </span><span class="z-source z-shell z-bash"><span class="z-punctuation z-definition z-compound z-braces z-begin z-shell">{</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell"><span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>articles<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span>:</span></span><span class="z-meta z-function-call z-arguments z-shell"> [</span> </span><span class="z-source z-shell z-bash"> <span class="z-punctuation z-definition z-compound z-braces z-begin z-shell">{</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell"><span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>title<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span>:</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>hello hyperstack<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span>,</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell"><span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>body<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span>:</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>EVH rocks<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span></span> </span><span class="z-source z-shell z-bash"> <span class="z-punctuation z-definition z-compound z-braces z-end z-shell">}</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">]</span></span> </span><span class="z-source z-shell z-bash"><span class="z-punctuation z-definition z-compound z-braces z-end z-shell">}</span> </span></code></pre> <p>And we’re done!</p> <p>Feel free to explore <a href="https://hyperstackjs.io/docs/getting-started/your-first-app">further in the docs</a>. You can play with <code>hyperstack routes</code> and <code>hyperstack portal</code> for the interactive REPL, and much much more.</p> <p><img src="https://rng0.io/blog/introducing-hyperstack-the-pragmatic-node-js-web-framework-for-builders/8.png" alt="Terminal window showing Hyperstack portal output" /></p> <h2 id="remember-complexity-always-finds-its-way">Remember: complexity always finds its way</h2> <blockquote> <p>We struggle with the complexities and avoid the simplicities — Norman Vincent Peale</p> </blockquote> <p>You can always choose to learn, and build complete frameworks from scratch. You can also invest time in building scalable architectures from scratch.</p> <p>But sometimes, you want to just build a product, or build something fun without subscribing to weeks of set up, infrastructure, and configuration as the preliminary step.</p> <p>For building products quickly, just use <a href="https://hyperstackjs.io/">Hyperstack</a>.</p> <h2 id="how-to-get-involved">How to Get Involved</h2> <p><img src="https://rng0.io/blog/introducing-hyperstack-the-pragmatic-node-js-web-framework-for-builders/9.png" alt="Rocky the puppy building servers" /></p> <p>You can start at the <a href="https://hyperstackjs.io/">website</a>, or jump into the <a href="https://github.com/hyperstackjs/hyperstack">Github repo</a> to read through the source code. We’re also accepting PRs if you have improvements to make or issues to report.</p> How to Do Pipeline Forensics, DFIR, or Just Find Files Quickly With Recon Sat, 19 Nov 2022 00:00:00 +0000 Unknown https://rng0.io/blog/how-to-do-pipeline-forensics-dfir-or-just-find-files-quickly-with-recon/ https://rng0.io/blog/how-to-do-pipeline-forensics-dfir-or-just-find-files-quickly-with-recon/ <h3 id="a-quick-introduction-to-the-open-source-tool">A quick introduction to the open-source tool</h3> <p><img src="https://rng0.io/blog/how-to-do-pipeline-forensics-dfir-or-just-find-files-quickly-with-recon/1.png" alt="" /></p> <p><a href="https://github.com/rusty-ferris-club/recon">Recon is an open-source tool</a> built in Rust 🦀, with great performance and ergonomics for finding files.</p> <p>You might want to:</p> <ul> <li>Run some validations or checks in your pipeline — for example, not producing binary files while building Javascript,</li> <li>Or you might want to run a digital forensics task on a specific host looking for a specific malware.</li> <li>Or, you might want to have a swiss-army knife for locating files using SQL.</li> </ul> <p><img src="https://rng0.io/blog/how-to-do-pipeline-forensics-dfir-or-just-find-files-quickly-with-recon/2.gif" alt="" /></p> <p>Check out the <a href="https://github.com/rusty-ferris-club/recon">Github repo here.</a></p> <h2 id="key-features">Key Features</h2> <ul> <li><strong>Query with SQL over files</strong> — find files using the full power of SQL queries</li> <li><strong>Find content with digests</strong> — use SHA256/512, md5, crc32 for duplicates, and other matchers for nontrivial matches to locate artifacts on hosts</li> <li><strong>Find malware or binaries with YARA</strong> — use YARA rules for matching against binary files efficiently</li> <li><strong>Finetune your search runtime</strong> — you can choose only the processing you need in order to cover more files quickly</li> <li><strong>Build your own scripts</strong> — pipe <code>recon</code> results to your own scripts with <code>--xargs</code></li> <li><strong>Export</strong> — use <code>--csv</code> or <code>--json</code>, or upload <code>recon.db</code> to your own servers for analysis</li> </ul> <h2 id="getting-recon">Getting Recon</h2> <p>Recon is free and open source. For macOS:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> brew tap rusty-ferris-club/tap</span> <span class="z-keyword z-operator z-logical z-and z-shell">&amp;&amp;</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">brew</span></span><span class="z-meta z-function-call z-arguments z-shell"> install recon</span> </span></code></pre> <p>Otherwise, grab a release from <a href="https://github.com/rusty-ferris-club/recon/releases">releases</a> and run <code>recon --help</code>:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> recon<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>help</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">SQL</span></span><span class="z-meta z-function-call z-arguments z-shell"> over files with security processing and tests</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Usage:</span></span><span class="z-meta z-function-call z-arguments z-shell"> recon <span class="z-keyword z-control z-regexp z-set z-begin z-shell">[</span>OPTIONS<span class="z-keyword z-control z-regexp z-set z-end z-shell">]</span></span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Options:</span></span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">-c,</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>config</span> <span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>CONFIG_FILE<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> Point to a configuration</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">-r,</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>root</span> <span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>ROOT<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> Target folder to scan</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">-q,</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>query</span> <span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>SQL<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> Query with SQL</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">-f,</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>file</span> <span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>DB_FILE<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> Use a specific DB file (file or :memory: for in memory</span><span class="z-meta z-function-call z-shell"></span>) <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">[default:</span></span><span class="z-meta z-function-call z-arguments z-shell"> recon.db]</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">-d,</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>delete</span> Clear data: delete existing cache database before running</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">-u,</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>update</span> Always walk files and update DB before query. Leave off to run query on existing recon.db.</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">-a,</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>all</span> Walk all files (dont consider .gitignore</span><span class="z-meta z-function-call z-shell"></span>) </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">--no-progress</span></span><span class="z-meta z-function-call z-arguments z-shell"> Don<span class="z-string z-quoted z-single z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&#39;</span>t display progress bars </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-string z-quoted z-single z-shell"> -m, --inmem Don<span class="z-punctuation z-definition z-string z-end z-shell">&#39;</span></span>t cache index to disk, run in-memory only</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">--xargs</span></span><span class="z-meta z-function-call z-arguments z-shell"> Output as xargs formatted list</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">--json</span></span><span class="z-meta z-function-call z-arguments z-shell"> Output as JSON</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">--csv</span></span><span class="z-meta z-function-call z-arguments z-shell"> Output as CSV</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">--no-style</span></span><span class="z-meta z-function-call z-arguments z-shell"> Output as a table with no styles</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">--fail-some</span></span><span class="z-meta z-function-call z-arguments z-shell"> Exit code failure if <span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span>some<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span> files are found</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">--fail-none</span></span><span class="z-meta z-function-call z-arguments z-shell"> Exit code failure if <span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span>no<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span> files are found</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">--verbose</span></span><span class="z-meta z-function-call z-arguments z-shell"> Show logs</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">-h,</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>help</span> Print help information</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">-V,</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>version</span> Print version information</span> </span></code></pre> <h2 id="a-better-find">A better ‘find’</h2> <p>Do you remember all of the arguments to <code>find</code>? Me neither. Recon simplifies finding files by letting you use SQL:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> recon<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>q</span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>select path,mode,uid from files where ext=&#39;rs&#39;<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">┌────────────────────────────┬────────────┬─────┐</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> path │ mode │ uid │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├────────────────────────────┼────────────┼─────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./xtask/src/main.rs │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-rw-r--</span> │ 501 │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├────────────────────────────┼────────────┼─────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./recon/tests/cli_tests.rs │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-r--r--</span> │ 501 │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├────────────────────────────┼────────────┼─────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./recon/src/os.rs │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-r--r--</span> │ 501 │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├────────────────────────────┼────────────┼─────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./recon/src/out.rs │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-r--r--</span> │ 501 │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├────────────────────────────┼────────────┼─────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./recon/src/bin/recon.rs │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-r--r--</span> │ 501 │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├────────────────────────────┼────────────┼─────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./recon/src/config.rs │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-r--r--</span> │ 501 │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├────────────────────────────┼────────────┼─────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./recon/src/lib.rs │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-r--r--</span> │ 501 │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├────────────────────────────┼────────────┼─────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./recon/src/data.rs │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-r--r--</span> │ 501 │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├────────────────────────────┼────────────┼─────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./recon/src/processing.rs │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-r--r--</span> │ 501 │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├────────────────────────────┼────────────┼─────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./recon/src/workflow.rs │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-r--r--</span> │ 501 │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├────────────────────────────┼────────────┼─────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./recon/src/matching.rs │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-r--r--</span> │ 501 │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">└────────────────────────────┴────────────┴─────┘</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">11</span></span><span class="z-meta z-function-call z-arguments z-shell"> of 233 files in 5.565875ms</span> </span></code></pre> <p>How about doing something with the results with the usual <code>xargs</code> combo?</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> recon<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>q</span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>select path from files where ext=&#39;rs&#39;<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>xargs</span></span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">xargs</span></span><span class="z-meta z-function-call z-arguments z-shell"> echo</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">./xtask/src/main.rs</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./recon/tests/cli_tests.rs ./recon/src/os.rs ./recon/src/out.rs ./recon/src/bin/recon.rs ./recon/src/config.rs ./recon/src/lib.rs ./recon/src/data.rs ./recon/src/processing.rs ./recon/src/workflow.rs ./recon/src/matching.rs</span> </span></code></pre> <h2 id="a-powerful-forensics-tool">A powerful forensics tool</h2> <p><strong>Recon can also be useful for security specialists</strong> trying to locate vulnerable binaries, or malware, and any other DFIR task at hand.</p> <p>You can <strong>configure processors and matchers</strong> in case you need some compute-heavy data to be available for query or matching such as:</p> <ul> <li>Digests or checksums (md5, SHAxxx, crc32)</li> <li>Path, content matching</li> <li>YARA rules</li> <li>For more, see <a href="https://github.com/rusty-ferris-club/recon/tree/master/examples">https://github.com/rusty-ferris-club/recon/tree/master/examples</a></li> </ul> <p><strong>Let’s add binary detection.</strong> Create a configuration file called <code>config.yaml</code>:</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">source</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">computed_fields</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">is_binary</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-constant z-language z-boolean z-yaml">true</span> </span></code></pre> <blockquote> <p><em>Did you know? The common way to detect if a file is binary, is to read a chunk of data from it (usually 1k bytes) and search for any non-text bytes in it. Largely: there’s no built-in OS metadata “flag” that indicates if a file is a binary file.</em></p> </blockquote> <p>We can use <code>-m</code> to avoid saving any cache to disk.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> recon<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>m</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>c</span> config.yaml<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>q</span> <span class="z-string z-quoted z-single z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&#39;</span>select path,is_binary,mode from files limit 6<span class="z-punctuation z-definition z-string z-end z-shell">&#39;</span></span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">┌───────────────┬───────────┬────────────┐</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> path │ is_binary │ mode │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├───────────────┼───────────┼────────────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./os.rs │ 0 │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-r--r--</span> │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├───────────────┼───────────┼────────────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./out.rs │ 0 │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-r--r--</span> │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├───────────────┼───────────┼────────────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./bin/main.rs │ 0 │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-r--r--</span> │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├───────────────┼───────────┼────────────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./config.rs │ 0 │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-r--r--</span> │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├───────────────┼───────────┼────────────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./lib.rs │ 0 │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-r--r--</span> │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├───────────────┼───────────┼────────────┤</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./recon.db │ 1 │<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rw-r--r--</span> │</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">└───────────────┴───────────┴────────────┘</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">6</span></span><span class="z-meta z-function-call z-arguments z-shell"> files in 145.81675ms</span> </span></code></pre> <h2 id="exploring-more">Exploring more</h2> <p>We have fully configured examples to explore and start from ( <a href="https://github.com/rusty-ferris-club/recon/pulls">open a PR</a> to add your own!):</p> <p><a href="https://github.com/rusty-ferris-club/recon/blob/master/examples/all-processors.yaml">all-processors.yaml</a>— Turn on all processors by default</p> <p><a href="https://github.com/rusty-ferris-club/recon/blob/master/examples/custom-walking.yaml">custom-walking.yaml</a>— Compute fields only for part of the walked directory tree</p> <p><a href="https://github.com/rusty-ferris-club/recon/blob/master/examples/file-classes.yaml">file-classes.yaml</a>— Configure and classify your own file classes</p> <p><a href="https://github.com/rusty-ferris-club/recon/blob/master/examples/find-log4shell.yaml">find-log4shell.yaml</a> — An example for finding the log4shell vulnerability using known digests</p> <p><a href="https://github.com/rusty-ferris-club/recon/blob/master/examples/using-yara.yaml">using-yara.yaml</a> — Using a simple YARA ruleset for matching</p> <p>Have a use case? want to contribute back? <a href="https://github.com/rusty-ferris-club/recon">feel free to open a PR</a></p> Tauri vs. Electron for Tray Apps Sat, 05 Nov 2022 00:00:00 +0000 Unknown https://rng0.io/blog/tauri-vs-electron-for-tray-apps/ https://rng0.io/blog/tauri-vs-electron-for-tray-apps/ <h3 id="build-a-desktop-tray-starter-app-using-tauri-rust-and-javascript">Build a desktop tray starter app using Tauri, Rust, and JavaScript</h3> <p><img src="https://rng0.io/blog/tauri-vs-electron-for-tray-apps/1.png" alt="" /></p> <p><a href="https://tauri.app/">Tauri</a> is similar to Electron in general. The principles are the same: build apps quickly by using JavaScript for doing most of the app / UI work.</p> <p>However, Tauri takes a radically different approach in how it is implemented, and therefore, achieves better results on virtually every possible criterion.</p> <h2 id="size-comparison">Size Comparison</h2> <p>We already know that there are two major pains with Electron-based apps, are:</p> <ul> <li>Bundle size (download size)</li> <li>RAM usage and growth over time</li> </ul> <p>With Electron, you start out with an 85MB bundle size for almost no functionality, and most apps come up to a 150MB download easily.</p> <p>In terms of memory usage, an Electron app starts out using 500MB of your RAM for doing almost nothing. This is why I don’t use any Electron app (so-called “native apps”) for various popular Web apps such as Slack and others. I just use Chrome, hoping it can optimize RAM usage across tabs in a better way.</p> <p>For Tauri: 8MB of bundle size for a whole lot of functionality, and peak memory of 150–200MB usage, most of it due to a browser being a browser. Browsers tend to just eat up memory (it uses the native Web view of each operating system).</p> <p>In terms of features, well, what took Electron years to get to (I’m talking as an Electron developer here) — you have it available right now in Tauri, in its first stable version. From a quality bundler and developer tooling, to a built-in updater, to tray icon support, and more.</p> <h2 id="design-and-architecture-comparison">Design and Architecture Comparison</h2> <p>An additional benefit you get from Tauri apps comes directly from its engineering mindset, which I’d guess Rust helped drive.</p> <h3 id="security-as-a-first-class-citizen">Security as a First-class Citizen</h3> <p><img src="https://rng0.io/blog/tauri-vs-electron-for-tray-apps/2.png" alt="" /></p> <p><a href="https://tauri.app/v1/references/architecture/security/">Source</a></p> <p>Security as a first-class citizen. Not many app platforms (actually: none that I know of) start out with a detailed description of their built-in security features, as well as an <a href="https://tauri.app/v1/references/architecture/security/">implementation of advanced features</a> for you to use, focusing on the JS side, the IPC, and more. This comes in addition to what Rust as a language provides in safety and stability. Even more impressive is the <a href="https://tauri.app/v1/references/security/">inclusion of threat modeling</a> and the thinking behind how to operate defensively: security by design.</p> <h3 id="performance-as-a-first-class-citizen">Performance as a First-class Citizen</h3> <p><img src="https://rng0.io/blog/tauri-vs-electron-for-tray-apps/3.png" alt="" /></p> <p><a href="https://tauri.app/v1/references/benchmarks/">Source</a></p> <p>Benchmarks as a first-class citizen. Again, not many platforms start out with this topic, providing highly detailed, <a href="https://tauri.app/v1/references/benchmarks/">automated and transparent benchmarking</a>. When this happens in a library, tool, or platform, you know that performance is a mainline KPI. And when you build apps, you want to ensure that the performance of the platform you’re using stays great so your app which is built on top of it stays great. If the platform has performance problems it automatically means your app has performance problems, and also — that you as a developer will probably have no way to resolve it.</p> <h3 id="complete-backend-and-frontend-separation">Complete backend and frontend separation</h3> <p><img src="https://rng0.io/blog/tauri-vs-electron-for-tray-apps/4.png" alt="" /></p> <p><a href="https://tauri.app/v1/references/architecture/process-model/">Source</a></p> <p>While this initially may be perceived as a disadvantage vs Electron, the fact that the backend process is built in Rust and the frontend is built in JavaScript is actually extremely beneficial:</p> <ul> <li>It forces us to really reason about the IPC, which encourages safety and stability</li> <li>In the case of Rust, it gives you exactly what you need in a backend process: performance, safety, stability, and a systems approach for when you need to operate tightly with the underlying system. You can’t really do that with a Node.js-based backend process (you’ll have to go native API in this case)</li> </ul> <p>And with how Tauri abstracts IPC with Rust macros and developer experience, even if you don’t know Rust, you’re going to be alright.</p> <h2 id="a-case-for-keeping-it-simple">A Case for Keeping it Simple</h2> <p>Building Electron apps is by no means simple. It’s virtually the peak of the paradox of choice, as with many JavaScript apps, but here — even more so. While you have the standard JavaScript fatigue in your frontend side of the Electron app, you have the same in the backend “main” process: you have to choose your JavaScript tooling, libraries, and principles, but limited as a pure Node.js process (so some libraries won’t work, and some would <em>only</em> work).</p> <p>With Tauri, the backend is written in Rust, and the Rust ecosystem is simple and mostly there’s one good library to do a thing. So you steer clear of the paradox of choice and have more time building.</p> <p>For the frontend part, you’re left with the same experience as with your Electron apps. You’re going to need to choose your bundler, app framework, styling framework, state management framework, linter, and more — whether you go with JavaScript or TypeScript.</p> <p>In that sense, building on Tauri is simpler.</p> <h2 id="how-to-build-a-tray-app-with-tauri">How to Build a Tray App With Tauri?</h2> <p><img src="https://rng0.io/blog/tauri-vs-electron-for-tray-apps/5.png" alt="" /></p> <p>For this exercise, let’s see what a “tray app” means. Off the bat, it’s actually many things. It’s a:</p> <ul> <li>Desktop app</li> <li>An operating system widget or accessory</li> <li>A long-living background process</li> <li>A windowing exercise (e.g. popping up at the correct place)</li> </ul> <p>And so, it might be a hard exercise for a new app platform to go through, which is perfect to understand how Tauri handles it. We already know many tray apps are implemented in Electron today, so how does Tauri fare?</p> <h3 id="example-case-docker-desktop-app">Example Case: Docker Desktop App</h3> <p>Let’s break down the Docker desktop app. We’re going to try and rebuild most of its parts in Tauri, and see what works.</p> <p>The Docker desktop requirement or spec is along the lines of:</p> <ul> <li>Tray with an updating icon (when Docker loads)</li> <li>Updating, dynamic menu items in tray, for example to show status “Docker is running” and current logged in user’s name when opening the tray menu</li> <li>A desktop window that opens (the Dashboard), which has a specific behavior:</li> <li>When showing the window (Tray → Open dashboard), the window should pop to your current workspace and monitor</li> <li>When clicking out of the window, or going to do something else, the window should hide</li> <li>This is a matter of taste, we can have the window hang around, and the user should explicitly hide it by clicking close (in fact, closing should quit, but we’re preventing that and hiding instead)</li> <li>The app should not have presence on your task bar / springboard. It should only live in your tray.</li> <li>This is a preference of taste, we can toggle it to be on/off.</li> <li>Has a specific size, cannot be resized</li> <li>Can be moved around, has a window shell (this can be removed, but let’s have the shell)</li> </ul> <p>In terms of infrastructure and mechanics, we need the following:</p> <ul> <li>Some kind of polling of status between the JavaScript app and the Rust main process. We can go:</li> <li>JavaScript polls Rust with setInterval and invoking a Tauri command, or</li> <li>Rust side pushing a periodic event by doing an <code>emit</code> on the main window for the JavaScript side to listen on</li> <li>Either way, we have a constant line of communication between both sides which is great.</li> <li>We’re turning a desktop app, into a long running powerful process.</li> <li>We also need ad-hoc calling code from JavaScript into Rust, this is magically solved by Tauri’s <code>command</code> abstraction</li> <li>Lastly, we need a fully-fledged desktop app framework and infrastructure. Here, we’ll use React, Chakra-UI, React Router, and Zustand or Redux for state and specifically state persistence (for example when storing user settings).</li> </ul> <p>You can go try the finished starter project here:</p> <p><a href="https://github.com/jondot/tauri-tray-app">https://github.com/jondot/tauri-tray-app</a></p> <h2 id="tauri-has-the-only-stable-system-tray-implementation">Tauri Has The Only Stable System Tray Implementation</h2> <p>This might come as a surprise, but compared to Go or Node.js, Rust hasn’t got a great system tray library, and Tauri actually has the best implementation right now. What we have is the following:</p> <ul> <li><a href="https://github.com/Ciantic/trayicon-rs">https://github.com/Ciantic/trayicon-rs</a> — works only on windows, and is designed to work with <code>winit</code></li> <li><a href="https://github.com/qdot/systray-rs">https://github.com/qdot/systray-rs</a> — not maintained anymore, and the author indicates it’s not an easy task</li> <li>Tauri made <a href="https://github.com/tauri-apps/tao">https://github.com/tauri-apps/tao</a> which implements a system tray functionality and is actually a fork of <a href="https://crates.io/crates/winit">https://crates.io/crates/winit</a></li> </ul> <h2 id="the-demo-app-overview">The Demo App: Overview</h2> <p><img src="https://rng0.io/blog/tauri-vs-electron-for-tray-apps/6.png" alt="" /></p> <p><a href="https://tauri.app/v1/references/architecture/inter-process-communication/">Source</a></p> <p>Set up and bring the starter app (you’ll need <code>pnpm</code> and you probably want to <a href="https://tauri.app/v1/guides/getting-started/prerequisites">run the getting started guide</a> to get the basic prereqs in place)</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> git clone https://github.com/jondot/tauri-tray-app</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cd tauri-tray-app</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> pnpm install</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo tauri dev</span> </span></code></pre> <h2 id="frontend">Frontend</h2> <p>Every Tauri app divides into frontend and the main process (Tauri core). In the starter project the frontend has the following major components:</p> <ul> <li><a href="https://vitejs.dev/">Vite</a> for building</li> <li>React as a UI framework</li> <li><a href="https://chakra-ui.com/">Chakra UI</a> as a component framework and for styling</li> <li><a href="https://github.com/remix-run/react-router">React Router</a> as a general purpose routing framework, for when an app has to load and unload “screens”</li> <li><a href="https://redux-toolkit.js.org/">Redux Toolkit</a> or <a href="https://github.com/pmndrs/zustand">Zustand</a> for state management and data persistence (both are wired, pick what you prefer)</li> </ul> <h2 id="main-process">Main process</h2> <p>Here are a few mechanics in our wishlist that we want in place in the core / main process part implemented in Rust and Tauri’s API.</p> <ul> <li>Avoiding closing from the window because its a tray app</li> <li>Hiding from taskbar</li> <li>Making sure the window appears in front of the user</li> <li>State — to be familiar we can use either of these (we’ll use state on the JavaScript side)</li> <li>JavaScript side (managing state as usual).</li> <li>Rust side — <a href="https://docs.rs/tauri/1.1.1/tauri/trait.Manager.html#method.state">the state manager</a> in Tauri can be used for general-purpose state storage.</li> <li>Background processes running and triggering</li> <li>JavaScript side — we’ll answer the question — will a naive setInterval loop live forever and ping the Rust core? (yes)</li> <li>And also what’s the best way to initiate such a forever loop?</li> <li>Controlling native features from the JavaScript side, such as setting a native menu item</li> </ul> <h2 id="setting-a-menu-item-dynamically-from-javascript">Setting a Menu Item Dynamically from JavaScript</h2> <p>This is easy to do, here’s the recipe for it:</p> <p>Create an item and give it a unique ID:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">add_item</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">CustomMenuItem<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>dynamic-item<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">to_string</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>Change me<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></code></pre> <p>Build a command, and request the app handle in the function signature:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">tauri</span>::<span class="z-variable z-annotation z-rust">command</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">set_menu_item</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">app_handle</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-path z-rust">tauri<span class="z-punctuation z-accessor z-rust">::</span></span>AppHandle, <span class="z-variable z-parameter z-rust">title</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">let</span> item_handle <span class="z-keyword z-operator z-assignment z-rust">=</span> app_handle<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">tray_handle</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">get_item</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>dynamic-item<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> item_handle<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">set_title</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>title</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">unwrap</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Register it using the following piece of code:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">invoke_handler</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">tauri<span class="z-punctuation z-accessor z-rust">::</span></span>generate_handler<span class="z-keyword z-operator z-logical z-rust">!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">[</span>greet<span class="z-punctuation z-separator z-rust">,</span> set_menu_item<span class="z-punctuation z-section z-group z-end z-rust">]</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></code></pre> <p>And call it from the JS side with invoke:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts"><span class="z-entity z-name z-function z-ts">setMenuItem</span></span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">invoke</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>set_menu_item<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">title</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-template z-ts"><span class="z-punctuation z-definition z-string z-template z-begin z-ts">`</span>count is <span class="z-meta z-template z-expression z-ts"><span class="z-punctuation z-definition z-template-expression z-begin z-ts">${</span></span><span class="z-meta z-template z-expression z-ts"><span class="z-meta z-embedded z-line z-ts"><span class="z-variable z-other z-readwrite z-ts">count</span></span><span class="z-punctuation z-definition z-template-expression z-end z-ts">}</span></span><span class="z-punctuation z-definition z-string z-template z-end z-ts">`</span></span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span></span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts">...</span> </span><span class="z-source z-ts"><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">button</span> <span class="z-variable z-other z-readwrite z-ts">onClick</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">setMenuItem</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-keyword z-operator z-relational z-ts">&gt;</span><span class="z-support z-class z-builtin z-ts">Set</span> <span class="z-variable z-other z-readwrite z-ts">menu</span> <span class="z-variable z-other z-readwrite z-ts">item</span><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">button</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></code></pre> <h2 id="long-running-processes">Long-running Processes</h2> <p>We have two ways to do this, in the sequence diagram below, both are charted out:</p> <p><img src="https://rng0.io/blog/tauri-vs-electron-for-tray-apps/7.png" alt="" /></p> <h3 id="long-running-processes-on-the-javascript-side">Long-running processes on the JavaScript side</h3> <p>We want to run a normal JavaScript <code>setInterval</code> and hope it stays alive and able to drive the Rust process as well through IPC (a Tauri <code>command</code>).</p> <p>In short, it works:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Note: setInterval is firing off a promise and does not wait for it to resolve.</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> depending on what you want to get done, it may be smarter to use a different</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> scheduling technique for an async call that may take longer than the interval</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> from time to time.</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">useEffect</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">interval</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-function z-ts">setInterval</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">invoke</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>interval_action<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">msg</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-template z-ts"><span class="z-punctuation z-definition z-string z-template z-begin z-ts">`</span>count is <span class="z-meta z-template z-expression z-ts"><span class="z-punctuation z-definition z-template-expression z-begin z-ts">${</span></span><span class="z-meta z-template z-expression z-ts"><span class="z-meta z-embedded z-line z-ts"><span class="z-variable z-other z-readwrite z-ts">count</span></span><span class="z-punctuation z-definition z-template-expression z-end z-ts">}</span></span><span class="z-punctuation z-definition z-string z-template z-end z-ts">`</span></span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-promise z-ts">then</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">s</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">any</span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">setMsg</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">s</span><span class="z-meta z-brace z-round z-ts">)</span> </span></span></span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">5000</span><span class="z-meta z-brace z-round z-ts">)</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span><span class="z-meta z-function-call z-ts"><span class="z-support z-function z-ts">clearInterval</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">interval</span><span class="z-meta z-brace z-round z-ts">)</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">count</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <p>But, beware of some pitfalls:</p> <ul> <li>Watch out for timers slowing down when the JavaScript part senses being in the background <a href="https://stackoverflow.com/questions/23506103/setinterval-slows-down-with-tab-window-inactive">https://stackoverflow.com/questions/23506103/setinterval-slows-down-with-tab-window-inactive</a></li> <li>Like always, something that’s async in an interval can drift. Since we don’t really have a way to hold up the interval if the async operation gets delayed for some reason</li> </ul> <h3 id="long-running-processes-on-the-rust-side">Long-running processes on the Rust side</h3> <p>We can have a long-running process on the Rust side, kick off a thread and have it do a forever loop with a small sleep between iterations.</p> <p>A few questions here:</p> <ul> <li>How and when to start it?</li> <li>What’s the concurrency model? since a Rust thread wants to ping the JavaScript side via IPC, there must be something to reason about here</li> </ul> <p>For starting out such a process, we can have a <code>command</code> and invoke it when we’re ready on the JavaScript side. There’s no right or wrong here, it’s a matter of what you need.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">tauri</span>::<span class="z-variable z-annotation z-rust">command</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">init_process</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">window</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-path z-rust">tauri<span class="z-punctuation z-accessor z-rust">::</span></span>Window</span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-path z-rust">std<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">thread<span class="z-punctuation z-accessor z-rust">::</span></span>spawn<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-storage z-modifier z-rust">move</span> <span class="z-keyword z-operator z-logical z-rust">||</span> <span class="z-keyword z-control z-rust">loop</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> window </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">emit</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>beep<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-support z-macro z-rust">format!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>beep: <span class="z-constant z-other z-placeholder z-rust">{}</span><span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-separator z-rust">,</span> <span class="z-meta z-path z-rust">chrono<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">Local<span class="z-punctuation z-accessor z-rust">::</span></span>now<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">to_rfc3339</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> </span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">unwrap</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-path z-rust">thread<span class="z-punctuation z-accessor z-rust">::</span></span>sleep<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">Duration<span class="z-punctuation z-accessor z-rust">::</span></span>seconds<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-constant z-numeric z-integer z-decimal z-rust">5</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">to_std</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">unwrap</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>Or, kick off a forever loop in the Tauri setup stage, where we still have access to an app. For emitting an event, we have to do so via a <code>Window</code>, since <code>App</code> won’t be thread-safe.</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">setup</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">|</span></span></span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-variable z-parameter z-rust">app</span><span class="z-punctuation z-section z-parameters z-end z-rust">|</span></span> </span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">let</span> window <span class="z-keyword z-operator z-assignment z-rust">=</span> app<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">get_window</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>main<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">unwrap</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-path z-rust">std<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">thread<span class="z-punctuation z-accessor z-rust">::</span></span>spawn<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-storage z-modifier z-rust">move</span> <span class="z-keyword z-operator z-logical z-rust">||</span> <span class="z-keyword z-control z-rust">loop</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> window </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">emit</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span> </span></span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>beep<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> <span class="z-support z-macro z-rust">format!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>beep: <span class="z-constant z-other z-placeholder z-rust">{}</span><span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-separator z-rust">,</span> <span class="z-meta z-path z-rust">chrono<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">Local<span class="z-punctuation z-accessor z-rust">::</span></span>now<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">to_rfc3339</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> </span></span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"> </span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">unwrap</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-macro z-rust">println!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>beep<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-path z-rust">thread<span class="z-punctuation z-accessor z-rust">::</span></span>sleep<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-path z-rust">Duration<span class="z-punctuation z-accessor z-rust">::</span></span>seconds<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-constant z-numeric z-integer z-decimal z-rust">5</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">to_std</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">unwrap</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></code></pre> <h2 id="running-async-commands">Running async commands</h2> <p>Let’s try kicking off a network request, which the Rust side performs. Here, we use request, which uses async, and <code>tokio</code> as the runtime.</p> <p>We know that this requires an initialized <code>tokio</code> runtime in any executable that we produce. As we’ll find out, Tauri uses <code>tokio</code>, so everything works out of the box. Just make the function async as you would normally, but remember: you <em>have to return a serializable error.</em></p> <p>To make it interesting we’re also returning a <code>Vec</code> (serializable) of <code>Content</code> (serializable).</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">tauri</span>::<span class="z-variable z-annotation z-rust">command</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust">async <span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">api_request</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">msg</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> <span class="z-meta z-function z-return-type z-rust"><span class="z-punctuation z-separator z-rust">-&gt;</span> <span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Result</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span><span class="z-meta z-generic z-rust"><span class="z-support z-type z-rust">Vec</span><span class="z-punctuation z-definition z-generic z-begin z-rust">&lt;</span>Content<span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span>, <span class="z-support z-type z-rust">String</span><span class="z-punctuation z-definition z-generic z-end z-rust">&gt;</span></span></span> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">let</span> res <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">reqwest<span class="z-punctuation z-accessor z-rust">::</span></span>get<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-string z-quoted z-double z-rust"><span class="z-punctuation z-definition z-string z-begin z-rust">&quot;</span>https://example.com<span class="z-punctuation z-definition z-string z-end z-rust">&quot;</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span>await </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">map_err</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">|</span></span></span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-variable z-parameter z-rust">e</span><span class="z-punctuation z-section z-parameters z-end z-rust">|</span></span> </span><span class="z-meta z-function z-closure z-rust">e<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">to_string</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">let</span> out <span class="z-keyword z-operator z-assignment z-rust">=</span> res<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">text</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span>await<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">map_err</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">|</span></span></span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-variable z-parameter z-rust">e</span><span class="z-punctuation z-section z-parameters z-end z-rust">|</span></span> </span><span class="z-meta z-function z-closure z-rust">e<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">to_string</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-keyword z-operator z-rust">?</span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-support z-type z-rust">Ok</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-support z-macro z-rust">vec!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">[</span>Content <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> body<span class="z-punctuation z-separator z-rust">:</span> out </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <p>After this completes, we’ll happily receive the serialized content as a JavaScript data object on the JS side.</p> <h2 id="navigation">Navigation</h2> <p>When navigating outside of a window with long-running interval, it is unloaded and cleared. And React Router works out of the box.</p> <ul> <li>Remember: we’re not really navigating in a website. So using a router like React Router becomes something of a specialized need. Instead of using a router, you can use some kind of Tabs control containing all of the screens of the app that you need.</li> </ul> <p>All in all, for long-running, processes its best to drop them in the react-router root, and for UI parts that change, drop them in an <code>&lt;Outlet/&gt;</code> in the root.</p> <h2 id="preventing-exit-and-hide-on-blur">Preventing Exit and Hide On Blur</h2> <p>The mix of pattern matching in Rust and Tauri’s API is just fantastic:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">on_window_event</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">|</span></span></span><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-variable z-parameter z-rust">event</span><span class="z-punctuation z-section z-parameters z-end z-rust">|</span></span> </span><span class="z-meta z-function z-closure z-rust"><span class="z-keyword z-control z-rust">match</span> event<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">event</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-path z-rust">tauri<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">WindowEvent<span class="z-punctuation z-accessor z-rust">::</span></span>CloseRequested <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> api<span class="z-punctuation z-separator z-rust">,</span> <span class="z-keyword z-operator z-range z-rust">..</span> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> <span class="z-keyword z-operator z-rust">=&gt;</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> don&#39;t kill the app when the user clicks close. this is important </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> event<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">window</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">hide</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">unwrap</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> api<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">prevent_close</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> <span class="z-meta z-path z-rust">tauri<span class="z-punctuation z-accessor z-rust">::</span></span><span class="z-meta z-path z-rust">WindowEvent<span class="z-punctuation z-accessor z-rust">::</span></span>Focused<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span><span class="z-constant z-language z-rust">false</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> <span class="z-keyword z-operator z-rust">=&gt;</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> hide the window automaticall when the user </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> clicks out. this is for a matter of taste. </span></span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> event<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">window</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">hide</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">unwrap</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> <span class="z-keyword z-operator z-rust">_</span> <span class="z-keyword z-operator z-rust">=&gt;</span> <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></span></span></span><span class="z-source z-rust"><span class="z-meta z-group z-rust"><span class="z-meta z-function z-closure z-rust"><span class="z-meta z-block z-rust"> </span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></code></pre> <h2 id="updating-the-native-menu">Updating The Native Menu</h2> <p>In general, you have a high degree of control over the native side of things. You can do whatever you want with an existing menu. However, adding or removing items gets complicated, because there’s no access to the current “state” of a menu.</p> <p>To add items, you need to rebuild the complete menu with the new item included, and re-set it via the native API:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> there&#39;s no way to grab the current menu, and add to it, creating </span></span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> an evergrowing menu. so, we rebuild the initial menu and add an item. </span></span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> this means we&#39;ll only add one item, but to add an arbitrary number, </span></span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> make this command accept an array of items. </span></span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> also, you probably would want to inject the new items in a specific place, </span></span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> so you&#39;d have to split the initial menu to [start] [your content] [end], </span></span><span class="z-source z-rust"><span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> where &#39;end&#39; contains things like &quot;show&quot; and &quot;quit&quot;. </span></span><span class="z-source z-rust"><span class="z-meta z-annotation z-rust"><span class="z-punctuation z-definition z-annotation z-rust">#</span><span class="z-punctuation z-section z-group z-begin z-rust">[</span><span class="z-variable z-annotation z-rust">tauri</span>::<span class="z-variable z-annotation z-rust">command</span><span class="z-punctuation z-section z-group z-end z-rust">]</span></span> </span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-function z-rust"><span class="z-storage z-type z-function z-rust">fn</span> </span><span class="z-entity z-name z-function z-rust">add_menu_item</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-begin z-rust">(</span><span class="z-variable z-parameter z-rust">app_handle</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-meta z-path z-rust">tauri<span class="z-punctuation z-accessor z-rust">::</span></span>AppHandle, <span class="z-variable z-parameter z-rust">id</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span>, <span class="z-variable z-parameter z-rust">title</span><span class="z-punctuation z-separator z-rust">:</span> <span class="z-keyword z-operator z-rust">&amp;</span><span class="z-storage z-type z-rust">str</span></span><span class="z-meta z-function z-rust"><span class="z-meta z-function z-parameters z-rust"><span class="z-punctuation z-section z-parameters z-end z-rust">)</span></span></span></span><span class="z-meta z-function z-rust"> </span><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">let</span> <span class="z-storage z-modifier z-rust">mut</span> menu <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-support z-function z-rust">build_menu</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> <span class="z-storage z-type z-rust">let</span> item <span class="z-keyword z-operator z-assignment z-rust">=</span> <span class="z-meta z-path z-rust">CustomMenuItem<span class="z-punctuation z-accessor z-rust">::</span></span>new<span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>id<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">to_string</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-separator z-rust">,</span> title</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> menu <span class="z-keyword z-operator z-assignment z-rust">=</span> menu<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">add_item</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>item</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"> app_handle<span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">tray_handle</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">set_menu</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>menu</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-accessor z-dot z-rust">.</span><span class="z-support z-function z-rust">unwrap</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span></span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span><span class="z-punctuation z-terminator z-rust">;</span> </span></span></span><span class="z-source z-rust"><span class="z-meta z-function z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span></span> </span></code></pre> <h3 id="conclusion">Conclusion</h3> <p>There’s so much more to Tauri. We’ve just touched on a few cases, which Tauri 1.1 handled very well — the platform is mature; more mature than where Electron was at a similar point in its evolution.</p> <p>While I haven’t yet pushed it to the official Apple store, you can build fully bundled and ready-to-distribute apps with Tauri today, and distribute them however you like.</p> <p>To build your next tray app, you can definitely start from <a href="https://github.com/jondot/tauri-tray-app">https://github.com/jondot/tauri-tray-app</a>. If you find that you can improve it, feel free to submit a pull request!</p> Running Rust Tasks With “xtask” and “xtaskops” Sat, 29 Oct 2022 00:00:00 +0000 Unknown https://rng0.io/blog/running-rust-tasks-with-xtask-and-xtaskops/ https://rng0.io/blog/running-rust-tasks-with-xtask-and-xtaskops/ <h3 id="pop-quiz-how-do-you-write-and-run-your-daily-tasks">Pop quiz: How do you write and run your daily tasks?</h3> <p><img src="https://rng0.io/blog/running-rust-tasks-with-xtask-and-xtaskops/1.png" alt="" /></p> <p>If you have a moderately large React project or a project in Go, or any other kind of language, you probably found yourself needing your own build, tests or CI tasks.</p> <p>For example, for <code>npm</code> or <code>yarn</code> in Node.js, you'd write your own <code>test:integration</code> task which run only your integration tests:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn test:integration</span> </span></code></pre> <p>And <code>test:integration</code> might be any series of shell commands which tell your test runner what to filter in.</p> <p>And sometimes, these jobs have dependencies, so you order them in the right way:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn clean</span> <span class="z-keyword z-operator z-logical z-and z-shell">&amp;&amp;</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">yarn</span></span><span class="z-meta z-function-call z-arguments z-shell"> build</span> <span class="z-keyword z-operator z-logical z-and z-shell">&amp;&amp;</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">yarn</span></span><span class="z-meta z-function-call z-arguments z-shell"> test:integration</span> </span></code></pre> <p>And then, you drop those into your <code>package.json</code> file:</p> <pre data-lang="json" class="language-json z-code"><code class="language-json" data-lang="json"><span class="z-source z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>ci<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span>: <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>yarn clean &amp;&amp; yarn build &amp;&amp; yarn test:integration<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span> </span></code></pre> <p>Which eventually become:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn ci</span> </span></code></pre> <p>For <code>make</code> or similar tools, for all kinds of tasks such as build, lint, test, coverage, CI, release, it looks pretty similar.</p> <p>Sometimes you have to write your own “CI tooling” shell scripts:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./ci/build-dev.sh</span> </span></code></pre> <h2 id="how-do-we-do-this-in-rust-projects">How do we do this in Rust projects?</h2> <p>It’s not really possible to do ad-hoc tasks with Cargo like it is with <code>yarn</code> or <code>npm</code>, by adding some shell script to your manifest file (for Rust - <code>cargo.toml</code>).</p> <p>People usually solve this with custom shell scripts invoked for a specific need, or by using (abusing?) <code>make</code>, similar to how the Go community did it.</p> <p>There are a few reasons why we’re going the easy route like this. They are:</p> <ul> <li>It’s easy, just name the task and go</li> <li>We all know shell commands</li> <li>Running order and failure modes are inherited from the shell, too</li> <li>Easy to experiment &amp; iterate</li> </ul> <h2 id="the-workflow-problem">The workflow problem</h2> <p><img src="https://rng0.io/blog/running-rust-tasks-with-xtask-and-xtaskops/2.png" alt="" /></p> <p>Trying out a shell command, and pasting it as a <code>make</code> task is great for simple cases.</p> <p>But, for any moderately large project, ask youself these questions:</p> <ul> <li>How do you supply custom parameters to <code>make</code>? (or <code>npm</code> tasks for that matter)</li> <li>How about supplying parameters to a shell script? and flags or options? and defaults?</li> </ul> <p>The answer is: you go back to what ever shell script voodoo <em>you</em> are familiar with at that moment.</p> <p>Prompting for user input is even worse, and running tasks in a specific order (resolving a graph of tasks) is left to you, hand-coding the order.</p> <h2 id="the-productivity-problem">The productivity problem</h2> <p><img src="https://rng0.io/blog/running-rust-tasks-with-xtask-and-xtaskops/3.png" alt="" /></p> <p>Like in “normal” software, to be productive you need to:</p> <ul> <li>Do some programmatic logic</li> <li>Reuse your existing codebase</li> <li>Import other people’s libraries or tasks</li> <li>Ability to debug, reason about, or test tasks</li> </ul> <p>All of these are not possible in <code>npm</code>, or <code>make</code> or similar tools, and it's a painfully bad experience with authoring shell scripts.</p> <h2 id="what-are-the-solutions-out-there">What are the solutions out there?</h2> <p><img src="https://rng0.io/blog/running-rust-tasks-with-xtask-and-xtaskops/4.png" alt="" /></p> <p>Obviously, there are a few solutions for this challenge.</p> <h3 id="declarative-linear-build-tools">Declarative “Linear” build tools</h3> <p>There are your typical tools that were grown to “run my shell command” such as <code>yarn</code>, <code>make</code>, etc.</p> <p>You get to use everything you know about environment variables, shell command ordering and logic ( <code>&amp;&amp;</code>, etc), use positional params <code>$1, $2,</code> etc. and hope for the best.</p> <h3 id="smarter-make-tools-semi-declarative">Smarter ‘make’ tools, semi-declarative</h3> <p>Tools like <code>just</code>, or <code>cargo-make</code> are purpose-built to solve task-related pains, which is a good thing.</p> <p>For example building tasks with <code>cargo-make</code> looks like this:</p> <pre data-lang="toml" class="language-toml z-code"><code class="language-toml" data-lang="toml"><span class="z-source z-toml"><span class="z-punctuation z-definition z-table z-begin z-toml">[</span><span class="z-meta z-tag z-table z-toml"><span class="z-entity z-name z-table z-toml">tasks</span><span class="z-punctuation z-separator z-table z-toml">.</span><span class="z-entity z-name z-table z-toml">A</span></span><span class="z-punctuation z-definition z-table z-end z-toml">]</span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">dependencies</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-punctuation z-definition z-array z-begin z-toml">[</span><span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>B<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span><span class="z-punctuation z-separator z-array z-toml">,</span> <span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>C<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span><span class="z-punctuation z-definition z-array z-end z-toml">]</span> </span></code></pre> <pre data-lang="toml" class="language-toml z-code"><code class="language-toml" data-lang="toml"><span class="z-source z-toml"><span class="z-punctuation z-definition z-table z-begin z-toml">[</span><span class="z-meta z-tag z-table z-toml"><span class="z-entity z-name z-table z-toml">tasks</span><span class="z-punctuation z-separator z-table z-toml">.</span><span class="z-entity z-name z-table z-toml">B</span></span><span class="z-punctuation z-definition z-table z-end z-toml">]</span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">dependencies</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-punctuation z-definition z-array z-begin z-toml">[</span><span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>D<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span><span class="z-punctuation z-definition z-array z-end z-toml">]</span> </span></code></pre> <pre data-lang="toml" class="language-toml z-code"><code class="language-toml" data-lang="toml"><span class="z-source z-toml"><span class="z-punctuation z-definition z-table z-begin z-toml">[</span><span class="z-meta z-tag z-table z-toml"><span class="z-entity z-name z-table z-toml">tasks</span><span class="z-punctuation z-separator z-table z-toml">.</span><span class="z-entity z-name z-table z-toml">C</span></span><span class="z-punctuation z-definition z-table z-end z-toml">]</span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">dependencies</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-punctuation z-definition z-array z-begin z-toml">[</span><span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>D2<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span><span class="z-punctuation z-definition z-array z-end z-toml">]</span> </span></code></pre> <pre data-lang="toml" class="language-toml z-code"><code class="language-toml" data-lang="toml"><span class="z-source z-toml"><span class="z-punctuation z-definition z-table z-begin z-toml">[</span><span class="z-meta z-tag z-table z-toml"><span class="z-entity z-name z-table z-toml">tasks</span><span class="z-punctuation z-separator z-table z-toml">.</span><span class="z-entity z-name z-table z-toml">D</span></span><span class="z-punctuation z-definition z-table z-end z-toml">]</span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">script</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>echo hello<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span> </span></code></pre> <pre data-lang="toml" class="language-toml z-code"><code class="language-toml" data-lang="toml"><span class="z-source z-toml"><span class="z-punctuation z-definition z-table z-begin z-toml">[</span><span class="z-meta z-tag z-table z-toml"><span class="z-entity z-name z-table z-toml">tasks</span><span class="z-punctuation z-separator z-table z-toml">.</span><span class="z-entity z-name z-table z-toml">D2</span></span><span class="z-punctuation z-definition z-table z-end z-toml">]</span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">alias</span></span><span class="z-punctuation z-definition z-key-value z-toml">=</span><span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>D<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span> </span></code></pre> <p>So we get:</p> <ul> <li>Task-first tooling with dependencies, order resolution</li> <li>Convenience for parameters and naming</li> </ul> <p>But, we’re ultimately limited by running commands on the shell, or the tool we’re using.</p> <h3 id="programmatic-language-dsl-tooling">Programmatic language DSL tooling</h3> <p>In this category, you have a library or an infrastructure, and you’re left to build tasks on your own.</p> <p>All you get is a way to run your fully-programming-language capable tasks in a convenient way, and some glue.</p> <p>A great example for this was <code>rake</code> in Ruby. It looked like <code>make</code> but you build all of your tasks in your native language: Ruby.</p> <p>At some point in time, a similar solution (which is more of a guideline) emerged for Rust: <code>xtask</code>.</p> <h2 id="xtask">xtask</h2> <p><img src="https://rng0.io/blog/running-rust-tasks-with-xtask-and-xtaskops/5.png" alt="" /></p> <p><code>xtask</code> is a convention, a project set up practice.</p> <p>If you follow the set up steps, you end up with a <code>cargo xtask</code> command and a new Rust project called <code>xtask</code> in which you build all your custom stuff.</p> <p>Generally using <code>xtask</code> means you use cargo as you're used to.</p> <p>It also means you are able to use Rust to its full potential: import libraries, reuse project logic, test, lint and run your build tasks safely.</p> <h2 id="setting-up-xtask">Setting up <code>xtask</code></h2> <p>We’re trying to move from a single crate project to a multiple crate workspace, where you have two members:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">[ your-project, xtask ] </span></code></pre> <p>If your project is called <code>acme</code> move your code to a new <code>acme/</code> inner folder.</p> <p>Then, within your project:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo new<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>bin</span> xtask</span> </span></code></pre> <p>Set up your workspace (root <code>cargo.toml</code>):</p> <pre data-lang="toml" class="language-toml z-code"><code class="language-toml" data-lang="toml"><span class="z-source z-toml"><span class="z-punctuation z-definition z-table z-begin z-toml">[</span><span class="z-meta z-tag z-table z-toml"><span class="z-entity z-name z-table z-toml">workspace</span></span><span class="z-punctuation z-definition z-table z-end z-toml">]</span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">members</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-punctuation z-definition z-array z-begin z-toml">[</span> </span><span class="z-source z-toml"> <span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>acme<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span><span class="z-punctuation z-separator z-array z-toml">,</span> </span><span class="z-source z-toml"> <span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>xtask<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span> </span><span class="z-source z-toml"><span class="z-punctuation z-definition z-array z-end z-toml">]</span> </span></code></pre> <p>Add an alias for <code>cargo</code> to know <code>xtask</code> (in <code>.cargo/config</code>):</p> <pre data-lang="toml" class="language-toml z-code"><code class="language-toml" data-lang="toml"><span class="z-source z-toml"><span class="z-punctuation z-definition z-table z-begin z-toml">[</span><span class="z-meta z-tag z-table z-toml"><span class="z-entity z-name z-table z-toml">alias</span></span><span class="z-punctuation z-definition z-table z-end z-toml">]</span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">xtask</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>run --package xtask --<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span> </span></code></pre> <p>That’s it. Now the <code>xtask</code> project is yours to build. Treat it like any other binary that should perform build tasks.</p> <p>This is where you’d want to:</p> <ul> <li>Take commands and parameters easily</li> <li>Route the commands to tasks your wrote easily</li> <li>Author tasks easily</li> </ul> <p>Right now, there’s no recommended way to author <code>xtask</code> tasks easily.</p> <p>This is why <code>xtaskops</code> was born.</p> <h2 id="using-xtaskops">Using xtaskops</h2> <p>Using <a href="https://github.com/jondot/xtaskops">xtaskops</a>, here is a fully loaded <code>xtask</code> binary, with more tasks than you'd need. It fits a single screen more or less:</p> <p><img src="https://rng0.io/blog/running-rust-tasks-with-xtask-and-xtaskops/6.png" alt="" /></p> <p>By using the <code>xtaskops</code> library, you get all of the common tasks in Rust for free, and then some cool ones too like <code>bloat-deps</code> (show biggest dependencies, by size) and <code>powerset</code> (runs tests over combinations of features).</p> <p>Just wire them to your CLI framework of choice (here, we’re using <code>clap</code>).</p> <p>Here’s some tasks that are available today in <code>xtaskops</code>:</p> <ul> <li>bloat_deps Show biggest crates in release build</li> <li>bloat_time Show crate build times</li> <li>dev Run cargo check followed by cargo test for every file change</li> <li>ci Run typical CI tasks in series: fmt, clippy, and tests</li> <li>coverage Run coverage</li> <li>docs Run cargo docs in watch mode</li> <li>install Instal cargo tools</li> <li>powerset Perform a CI build with powerset of features</li> </ul> <p>For running your own commands, use the <code>cmd!</code> macro:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-support z-macro z-rust">cmd!</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-begin z-rust">(</span>your<span class="z-punctuation z-separator z-rust">,</span>command<span class="z-punctuation z-separator z-rust">,</span>here</span><span class="z-meta z-group z-rust"><span class="z-punctuation z-section z-group z-end z-rust">)</span></span> </span></code></pre> <h2 id="using-xtask-for-every-project">Using <code>xtask</code> for every project</h2> <p>Instead of manually setting up <code>xtask</code> over and over for every project, you can use <a href="https://github.com/rusty-ferris-club/rust-starter">rust-starter</a> which is <code>xtask</code>-driven.</p> <p>For existing projects, you can copy the <code>xtask</code> files or use <a href="https://github.com/rusty-ferris-club/backpack">Backpack</a> which can grab parts of repositories easily.</p> <p>Of course, you may want to make a starter project of your own: set up <code>xtask</code> once for yourself, and reuse your project for all of your work. Then use <a href="https://github.com/rusty-ferris-club/backpack">Backpack</a> to kick off every new project from your repository address.</p> <h2 id="conclusion">Conclusion</h2> <p>Before you add another <code>make</code> task, or write another custom shell build script, remember:</p> <ul> <li>Your build, ci, workflow, errands code — is still maintained code</li> <li>Don’t underestimate maintenance overhead for a task using shell commands</li> <li>Your shell voodoo better be bug free, tested, and reasoned about</li> <li>It’s better to reuse team investment in knowledge, code, standards, and practices</li> <li><code>make</code> can be tricky and require forgotten knowledge (e.g. <code>.PHONY</code>)</li> <li>package.json tasks often require additional packages ( <code>rimraf</code>, <code>concurrently</code>)</li> </ul> <p>You can use <code>xtask</code> to alleviate all these pains, and write more Rust code.</p> Be More Productive with Starter Projects and Backpack Tue, 12 Jul 2022 00:00:00 +0000 Unknown https://rng0.io/blog/be-more-productive-with-starter-projects-and-backpack/ https://rng0.io/blog/be-more-productive-with-starter-projects-and-backpack/ <blockquote> <p>tl;dr. I’ve built <a href="https://github.com/rusty-ferris-club/backpack">backpack</a>, a tool written in Rust that helps you use starter and template projects. I also explain what are the TEA and TEAM frameworks for productivity and how to manage Time, Energy and Attention with starter projects.</p> </blockquote> <p><a href="https://github.com/topics/starter">Stater projects</a> are k <strong>ey to unlocking your productivity</strong>, for side-projects and hobbies, or experiments at work.</p> <p>There’s value in creating your own <strong>starter projects from time to time</strong>, to learn some new technology or technique, and at the same time, <strong>using a public starter project to launch quickly</strong>.</p> <p>From a productivity point of view, I often use the TEA (Time, Energy, Attention) framework or the TEAM (Time, Energy, Attention, and Motivation) framework.</p> <h2 id="what-is-the-tea-team-framework">What is the TEA / TEAM framework?</h2> <p><strong>TEA and TEAM are acronyms to a framework that you can use to evaluate and create effective work for yourself or your team.</strong></p> <p>They stand for the what you should optimize for executing effectively:</p> <ul> <li>T — Time</li> <li>E — Energy</li> <li>A — Attention</li> <li>M — Motivation</li> </ul> <h2 id="time">Time</h2> <p>Use the 3 W’s: <strong>When, Wait, Worry</strong> when you’re planning your tasks and you want to figure out timing so that you feel comfortable with that specific task.</p> <p>The idea is to <strong>put a system in place</strong> that will control and regulate our time investment and commitment.</p> <p>To explain intuitively, <strong>think like a customer</strong> ordering a product. Ask these questions:</p> <ul> <li><em>(W)hen do I get it?</em></li> <li><em>How long do I (W)ait?</em></li> <li><em>When do I need to (W)orry about my order?.</em></li> </ul> <p>For example, if I order a Pizza now, I know I should get it more or less at the time for my favorite TV show today, it should take 30 minutes, and if I don’t get it within 60 minutes I need to call the place and see what’s going on.</p> <p><strong>Let’s convert this to a tech task</strong>. Let’s say we want to add new functionality to our CI so that it will publish an npm package automatically. We know that building it should take around 30 minutes: get some Github Action, drop it into our workflow, set up some permissions, and then spend the rest of the time kicking the tires seeing that it works. That’s the Wait part.</p> <p><strong>But, if this becomes a 3 hour rabbit hole because there’s too much unknowns we didn’t expect</strong> (as often is with Github Actions), we know that we need to worry. So let’s set that as the time to raise a flag, stop, and park the task.</p> <p>Next, what we want to do is decide when to do it, or when it should be at the “done” state. <strong>If you know you’re doing CI work on Wednesday, then that’s when.</strong></p> <p>To nail time, we’ve focused on the <strong>3 W’s: When, Wait, Worry.</strong> Under the hood, we’ve sorted out:</p> <ul> <li><strong>Scope</strong>: made sure we scope the task with a manageable chunk of time (30m), so: not too big.</li> <li><strong>Flagging system</strong>: we outlined when to hard stop. Often a waste of time because we’re developing a tunnel vision — plowing straight ahead when the smarter thing to do is to stop and recalculate.</li> <li><strong>Prioritization and commitment</strong>: by saying when, we’ve sorted out priority and committed ourselves to a goal.</li> </ul> <h2 id="energy">Energy</h2> <p><strong>Energy</strong> is about feeling empowered to complete your goals, and being able to understand your personal levels of energy through out your day.</p> <p>It’s <strong>about hacking the system</strong>, and finding ways to boost your energy when you’re done, and to recognize and make use of your <strong>high energy levels</strong> when you’re feeling great.</p> <p><strong>The first step is to be mindful of when you’re not feeling great about your energy.</strong> The rest, is personal. As an example, for me, I take a short walk to boost my energy levels in mornings, and I make sure I have a system like GTD (Getting Things Done) as my own productivity ritual, but YMMV.</p> <h2 id="attention">Attention</h2> <p><strong>Attention</strong> is about your ability to stay focused through out the day. There are different ways to get at it, but often it’s about:</p> <ul> <li><strong>Turning off IMs,</strong> interruptions</li> <li><strong>Blocking out time,</strong> through calendar or otherwise</li> <li><strong>Placing yourself in a different environment</strong> (I like coffee shops)</li> <li>Getting to <strong>a “flow” state of mind</strong>(I like specific kinds of music)</li> </ul> <h2 id="motivation">Motivation</h2> <p><strong>Motivation</strong> is about making sure you know where you’re headed. Writing down the end goal to yourself, keeping some mantras written down, are the best way to remind yourself why you’re doing what you’re doing.</p> <p><strong>A good idea is to also revisit this from time to time</strong>, and to kill off projects that you’re not feeling great about anymore — it happens.</p> <h2 id="the-destroyer-of-team-starting-up-a-new-project">The destroyer of TEAM: starting up a new project</h2> <p>Starting up a project may be exciting for some, but for those who have a clear goal and motivation, it may be a killer of productivity. Typically when starting up a project you need to:</p> <ul> <li><strong>Pick the tech stack</strong>, weigh in alternatives: paradox of options big time.</li> <li><strong>Start building the tech stack</strong>, hit roadblocks and the reality of your previous choices</li> <li><strong>Repeat until you have a stable starter project</strong></li> <li><strong>Implement all the boilerplate, infrastructure, policies</strong></li> </ul> <p>Although it may have been an exciting experience if this is your first time dealing with tech stack X or Y, by the time you’re done, you’re out of energy, you’ve depleted your time allowance, and your motivation has suffered a big blow.</p> <h2 id="how-do-starter-projects-help-maintain-team">How do starter projects help maintain TEAM</h2> <p><strong>To work around this, I developed a habit of extracting starter projects from my most successful projects.</strong> I then made them into a repo, and created a script that clones, cleans up, and sets up a new project from that starter skeleton.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> ls <span class="z-meta z-group z-expansion z-tilde"><span class="z-variable z-language z-tilde z-shell">~</span></span>/.dotfiles/bin/init-<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">/Users/jondot/.dotfiles/bin/init-cpp</span></span><span class="z-meta z-function-call z-arguments z-shell"> /Users/jondot/.dotfiles/bin/init-node-babel /Users/jondot/.dotfiles/bin/init-react-native-comp /Users/jondot/.dotfiles/bin/init-ts-lib</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">/Users/jondot/.dotfiles/bin/init-cpp-conan</span></span><span class="z-meta z-function-call z-arguments z-shell"> /Users/jondot/.dotfiles/bin/init-node-cli /Users/jondot/.dotfiles/bin/init-rust-cli /Users/jondot/.dotfiles/bin/init-webpack-lib</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">/Users/jondot/.dotfiles/bin/init-go-cli</span></span><span class="z-meta z-function-call z-arguments z-shell"> /Users/jondot/.dotfiles/bin/init-node-lib /Users/jondot/.dotfiles/bin/init-rust-lib /Users/jondot/.dotfiles/bin/init-webpack-sls</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">/Users/jondot/.dotfiles/bin/init-jvm-proj-gradle</span></span><span class="z-meta z-function-call z-arguments z-shell"> /Users/jondot/.dotfiles/bin/init-py /Users/jondot/.dotfiles/bin/init-scala-gradle</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">/Users/jondot/.dotfiles/bin/init-node</span></span><span class="z-meta z-function-call z-arguments z-shell"> /Users/jondot/.dotfiles/bin/init-react-component /Users/jondot/.dotfiles/bin/init-scala-sls-gradle</span> </span></code></pre> <p>It’s funny, that <code>bin</code> folder tells my own personal story of self development over the last 10 years.</p> <p><strong>Here’s one of these init scripts:</strong></p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cat <span class="z-meta z-group z-expansion z-command z-backticks z-shell"><span class="z-punctuation z-section z-group z-begin z-shell">`</span><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">which</span></span><span class="z-meta z-function-call z-arguments z-shell"> /Users/jondot/.dotfiles/bin/init-cpp</span><span class="z-punctuation z-section z-group z-end z-shell">`</span></span></span> </span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell">!/bin/sh</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">git</span></span><span class="z-meta z-function-call z-arguments z-shell"> clone https://github.com/jondot/init-cpp <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">1</span></span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-support z-function z-cd z-shell">cd</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">1</span></span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">mkdir</span></span><span class="z-meta z-function-call z-arguments z-shell"> build</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-support z-function z-cd z-shell">cd</span></span><span class="z-meta z-function-call z-arguments z-shell"> build</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">cmake</span></span><span class="z-meta z-function-call z-arguments z-shell"> ../</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">make</span></span> </span></code></pre> <p>With time this became a reflex. <strong>Every time I wanted to start a project, I say</strong> <code>init-foo</code> <strong>,</strong> and I have a pristine, fully rigged with best practices starter project.</p> <p><strong>This helped me optimize TEAM:</strong></p> <ul> <li><strong>Time</strong>: by using a starter, I’ve de-risked time. The choices that are already made there gives me smooth sailing. I can guesstimate better, and commit safely.</li> <li><strong>Energy</strong>: by not having so many choices, I’m keeping my energy fresh. The first line of code I write — is a business logic line of code.</li> <li><strong>Attention</strong>: again, not reading about “X vs Y”, and “how to infrastructure X”, keeps my attention very clearly on my goal: building something useful.</li> <li><strong>Motivation</strong>: getting a starter project that allows for saying <code>yarn start</code> and you have a working, live, app is a motivation boost.</li> </ul> <h2 id="the-missing-part-supercharge-the-way-you-use-starter-projects">The missing part: supercharge the way you use starter projects</h2> <p>I started creating these little binaries over 10 years ago, but meanwhile, Github has supported a <code>template</code> repository type, which is what I did, and, smarter and brighter people have created a ton of starter and template projects.</p> <p><strong>I found that I needed a small tool to use different templates easily, so I built</strong> <code>backpack</code> <strong>.</strong> <code>backpack</code> is a super tool for using starter projects and templates directly from Github repos. If you want to use a starter project you do this:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> bp new kriasoft/react-starter-kit my-app</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">🔮</span></span><span class="z-meta z-function-call z-arguments z-shell"> Resolving...</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">🚚</span></span><span class="z-meta z-function-call z-arguments z-shell"> Fetching content...</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">🎒</span></span><span class="z-meta z-function-call z-arguments z-shell"> Unpacking files...</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">🎉</span></span><span class="z-meta z-function-call z-arguments z-shell"> Done in: <span class="z-string z-quoted z-single z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&#39;</span>my-app<span class="z-punctuation z-definition z-string z-end z-shell">&#39;</span></span></span> </span></code></pre> <p><strong>How is this different from a</strong> <code>git clone</code> <strong>?</strong></p> <ul> <li><strong>It’s faster</strong>(takes tarball by default or Git without history)</li> <li><strong>Caches content</strong>, because people reuse starters (next time you <code>new</code> a project, it'll be from local cache)</li> <li><strong>You can apply partial content</strong> to your project, such as specific folders, for example to copy the CI configuration, do <code>bp apply kriasoft/react-starter-git/-/.github</code></li> <li><strong>Allows to set up aliases</strong> so you can do <code>bp new react my-app</code> which will map to <code>kriasoft/react-starter-kit</code> in your own project.</li> <li><strong>You can use shortlinks</strong> like <code>gl:foo/bar</code> for Gitlab, or <code>gh-acme:foo/bar</code> for your own Github deployed instance</li> <li><strong>You can mix and match:</strong> get a branch or tag, and from that a subfolder, on a custom hosted Gitlab: <code>bp apply my-gl:foo/infra#v2.0.0/-/terraform</code>, and map that to an alias: <code>bp apply team-terraform</code>.</li> <li><strong>And on the roadmap:</strong> custom hooks (automating install and set up copy), and variable swaps (to rename tokens in files)</li> </ul> <h2 id="getting-started">Getting started</h2> <p>For macOS:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">brew</span></span><span class="z-meta z-function-call z-arguments z-shell"> tap rusty-ferris-club/tap</span> <span class="z-keyword z-operator z-logical z-and z-shell">&amp;&amp;</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">brew</span></span><span class="z-meta z-function-call z-arguments z-shell"> install backpack</span> </span></code></pre> <p>Otherwise, grab a release from <a href="https://github.com/rusty-ferris-club/backpack/releases">releases</a>.</p> <h2 id="using-a-starter-project">Using a starter project</h2> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> bp new <span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>shortlink<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> <span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>destination<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> </span></span></code></pre> <p>A shortlink is a pointer to a Git repo which looks like this:</p> <p><img src="https://rng0.io/blog/be-more-productive-with-starter-projects-and-backpack/1.png" alt="" /></p> <p>Any one of these is a legal shortlink:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">user/repo -&gt; resolves to https://github.com/user/repo </span><span class="z-text z-plain">gl:user/repo -&gt; resolves to https://gitlab.org/user/repo </span><span class="z-text z-plain">user/repo/-/subfolder -&gt; takes only &#39;subfolder&#39; </span><span class="z-text z-plain">user/repo#wip -&gt; takes the &#39;wip&#39; branch </span></code></pre> <p>So this will use a <a href="https://github.com/kriasoft/react-starter-kit">React starter</a>:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> bp new kriasoft/react-starter-kit my-app</span> </span></code></pre> <h2 id="creating-your-own-list-of-projects">Creating your own list of projects</h2> <p>You can configure your own list of favorite starters:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> bp config<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>init</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>global</span></span> </span></code></pre> <p>And then edit your <code>aliases</code>:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> vim <span class="z-meta z-group z-expansion z-tilde"><span class="z-variable z-language z-tilde z-shell">~</span></span>/.backpack/backpack.yaml</span> </span></code></pre> <p>Example:</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">aliases</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">rust</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">shortlink</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">rusty-ferris-club/rust-starter</span> <span class="z-comment z-line z-number-sign z-yaml"><span class="z-punctuation z-definition z-comment z-line z-number-sign z-yaml">#</span> you can use any custom prefix here too </span></span><span class="z-source z-yaml"> <span class="z-comment z-line z-number-sign z-yaml"><span class="z-punctuation z-definition z-comment z-line z-number-sign z-yaml">#</span> is_git: true # fetch from ssh for private repos </span></span></code></pre> <p>And then this will always work for your user:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> bp new rust my-project</span> </span></code></pre> <h2 id="copying-parts-of-a-starter-or-a-repo">Copying parts of a starter or a repo</h2> <p>If you’d like to copy <strong>parts of a project</strong>, you can with the <code>/-/</code> separator. For example, to get the CI workflow and setup you can do this:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> bp apply rusty-ferris-club/rust-starter/-/.github</span> </span></code></pre> <p>It will apply the necessary <code>.github</code> files onto your existing project.</p> <h2 id="sharing-starter-projects-with-your-team">Sharing starter projects with your team</h2> <p>You can create a local configuration for a repo, s <strong>o that you can check it in with the rest of the project:</strong></p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell"><span class="z-meta z-group z-expansion z-tilde"><span class="z-variable z-language z-tilde z-shell">~</span></span>$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cd logs-service</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">logs-service$</span></span><span class="z-meta z-function-call z-arguments z-shell"> bp config<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>init</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">wrote:</span></span><span class="z-meta z-function-call z-arguments z-shell"> .backpack.yaml</span> </span></code></pre> <p>Now <strong>anyone working in the</strong> <code>logs-service</code> directory will share the configuration. <code>backpack</code> first looks for local configuration, and then merges it with your global configuration as well.</p> <p>So, for example, you can share your custom hosted team Gitlab and set it as a default:</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">vendors</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">default</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">kind</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">gitlab</span> <span class="z-comment z-line z-number-sign z-yaml"><span class="z-punctuation z-definition z-comment z-line z-number-sign z-yaml">#</span> options: gitlab | github | bitbucket </span></span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">base</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">my.gitlab.com</span> </span><span class="z-source z-yaml"><span class="z-comment z-line z-number-sign z-yaml"><span class="z-punctuation z-definition z-comment z-line z-number-sign z-yaml">#</span> custom: </span></span><span class="z-source z-yaml"><span class="z-comment z-line z-number-sign z-yaml"><span class="z-punctuation z-definition z-comment z-line z-number-sign z-yaml">#</span> # custom github org to prefix, and also overrides the &#39;gh:&#39; prefix. </span></span><span class="z-source z-yaml"><span class="z-comment z-line z-number-sign z-yaml"><span class="z-punctuation z-definition z-comment z-line z-number-sign z-yaml">#</span> # $ backpack new gh:my-repo my-repo </span></span><span class="z-source z-yaml"><span class="z-comment z-line z-number-sign z-yaml"><span class="z-punctuation z-definition z-comment z-line z-number-sign z-yaml">#</span> gh: </span></span><span class="z-source z-yaml"><span class="z-comment z-line z-number-sign z-yaml"><span class="z-punctuation z-definition z-comment z-line z-number-sign z-yaml">#</span> kind: github </span></span><span class="z-source z-yaml"><span class="z-comment z-line z-number-sign z-yaml"><span class="z-punctuation z-definition z-comment z-line z-number-sign z-yaml">#</span> base: github.com/my-org </span></span><span class="z-source z-yaml"><span class="z-comment z-line z-number-sign z-yaml"><span class="z-punctuation z-definition z-comment z-line z-number-sign z-yaml">#</span> </span></span></code></pre> <p>And now:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">logs-service$</span></span><span class="z-meta z-function-call z-arguments z-shell"> bp some/project</span> </span></code></pre> <p>Will connect to your team’s <code>my.gitlab.com</code> by default.</p> <h2 id="summary">Summary</h2> <p><strong>Using</strong> <code>backpack</code> <strong>is like creating your own backpack with your tools and equipment in it.</strong> When ever you want to create a new project or to grab parts of an existing one, <code>backpack</code> is there: <code>bp new</code> or <code>bp apply</code>.</p> <p>Now that the foundations are there, we plan to build some more exciting features on top of <code>backpack</code> to automate the boring and energy depleting task of setting up new projects:</p> <ul> <li><strong>Hooks</strong>: run custom commands to set up a project after <code>new</code></li> <li><strong>Swaps</strong>: given a dictionary of replacements, and target locations — swap tokens such as name of project, name of author and others, right after <code>new</code> or <code>apply</code>.</li> </ul> How to do code coverage in Rust Sat, 09 Jul 2022 00:00:00 +0000 Unknown https://rng0.io/blog/how-to-do-code-coverage-in-rust/ https://rng0.io/blog/how-to-do-code-coverage-in-rust/ <p>Now that we have <a href="https://blog.rust-lang.org/2022/04/07/Rust-1.60.0.html#source-based-code-coverage">source based coverage</a> in Rust, we’re going to look at how to do the following:</p> <ul> <li>Run coverage locally</li> <li>Integrate with IDE for inline coverage indicators</li> <li>Run coverage in CI and upload results to a coverage service provider</li> </ul> <p>But first, let’s look at the history of code coverage in Rust, it might help you when you Google around for solutions, to understand what’s current and what’s not.</p> <h2 id="history-of-code-coverage-in-rust">History of code coverage in Rust</h2> <h2 id="tarpaulin">Tarpaulin</h2> <p><a href="https://github.com/xd009642/tarpaulin">Tarpaulin</a> is an easy tool to run code coverage with Rust, but has limitations. An intuitive description for <a href="https://xd009642.github.io/2019/10/02/Tarpaulin-and-the-futures.html">how Tarpaulin works</a>: it will <strong>instrument your code</strong> and then use <code>ptrace</code> to eavesdrop on what's happening in order to count lines for coverage analysis.</p> <p>This is why Tarpaulin <strong>only supports Linux</strong>, and also specifically <strong>only supports x86_64 processors</strong>.</p> <p>For accuracy’s sake — Tarpaulin provides line coverage and is fairly reliable but still include some inaccuracies in the results.</p> <h2 id="gcov">gcov</h2> <p><strong>Before source based coverage</strong>, Rust relied on the <code>gcov</code> technique, which relied on <strong>debug info</strong> to map from LLVM IR (bottom, generated and hidden from you) to lines of source code (top, what you're used to seeing). Think about it like making a really good guess <strong>for <em>names</em> into lines of code</strong>, and keeping count of which line is ran and how many times.</p> <p>As such, and since the <strong>resolution of information is lost</strong> from layer to layer as you move from source to compiled binary form, the <code>gcov</code> based techinque was not the most accurate.</p> <h2 id="source-based-coverage">Source based coverage</h2> <p><a href="https://blog.rust-lang.org/2022/04/07/Rust-1.60.0.html#source-based-code-coverage"><strong>Source based coverage</strong></a> lets Rust have instrumentation done right at the compiler level while then generating IR from source, which moves down to all compilation stages, and then <strong>lets the coverage analysis tools use all of that high-quality information</strong> to correlate and pinpoint exactly a given low-level construct to its higher-level source code (that you wrote).</p> <p>This results in a <strong>much higher accuracy level</strong> and better implementation of coverage instrumentation. To get a feel of what “accuracy is”, then take a look at this short-circuit statement:</p> <pre data-lang="rust" class="language-rust z-code"><code class="language-rust" data-lang="rust"><span class="z-source z-rust"><span class="z-keyword z-control z-rust">if</span> a <span class="z-keyword z-operator z-comparison z-rust">&lt;</span> b <span class="z-keyword z-operator z-logical z-rust">&amp;&amp;</span> c <span class="z-keyword z-operator z-comparison z-rust">&gt;</span> d <span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-begin z-rust">{</span> </span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"> <span class="z-comment z-line z-double-slash z-rust"><span class="z-punctuation z-definition z-comment z-rust">//</span> .. </span></span></span><span class="z-source z-rust"><span class="z-meta z-block z-rust"></span><span class="z-meta z-block z-rust"><span class="z-punctuation z-section z-block z-end z-rust">}</span></span> </span></code></pre> <p>If <code>a &gt; b</code> (the <strong>inverse</strong> of the first condition), we know that the <strong>next condition does not need to be run</strong>. This is shortcircuiting. And so, source based coverage will be able to tell you <strong>that</strong> <code>c &gt; d</code> <strong>was never exercised</strong>, so you could write a test for that condition as well, where the gcov method would list all of that line as exercised (!).</p> <p>Today, <strong>source based coverage</strong> in Rust is the recommended way to do coverage. It’s also the most usable, most accurate, and most portable way to do coverage.</p> <h2 id="our-goals">Our goals</h2> <p>Source based coverage opens up the opportunity to have a <strong>holy-trinity of coverage</strong>:</p> <ul> <li>Run locally for <strong>fast feedback</strong> for an individual.</li> <li>Seamless <strong>inline</strong> IDE integration</li> <li>Run in CI, PRs, and centrally <strong>for a team</strong></li> </ul> <h2 id="running-locally">Running Locally</h2> <p>To do code coverage with the new <strong>source based coverage</strong> features, you need to perform a few moves. <strong>We’ll see the manual steps first, and automate it all later</strong>.</p> <p>Generally we want <strong>tests to drive our coverage</strong>, so we’ll run <code>cargo test</code>, but we need to enable instrumentation, and also tell LLVM to output its raw profiling data.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> CARGO_INCREMENTAL=0 RUSTFLAGS=<span class="z-string z-quoted z-single z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&#39;</span>-Cinstrument-coverage<span class="z-punctuation z-definition z-string z-end z-shell">&#39;</span></span> LLVM_PROFILE_FILE=<span class="z-string z-quoted z-single z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&#39;</span>cargo-test-%p-%m.profraw<span class="z-punctuation z-definition z-string z-end z-shell">&#39;</span></span> cargo test</span> </span></code></pre> <p>Next, we need to take all of that data, and <strong>generate a report</strong> that can be ingested by tools and read by machines, or that we can browse around as humans.</p> <p><strong>As HTML:</strong></p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> grcov .<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>binary-path</span> ./target/debug/deps/<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>s</span> .<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>t</span> html<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>branch</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>ignore-not-existing</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>ignore</span> <span class="z-string z-quoted z-single z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&#39;</span>../*<span class="z-punctuation z-definition z-string z-end z-shell">&#39;</span></span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>ignore</span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>/*<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>o</span> target/coverage/html</span> </span></code></pre> <p><img src="https://rng0.io/blog/how-to-do-code-coverage-in-rust/1.jpg" alt="" /></p> <p><strong>As</strong> <code>lcov</code> <strong>, a popular output format:</strong></p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> grcov .<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>binary-path</span> ./target/debug/deps/<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>s</span> .<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>t</span> lcov<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>branch</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>ignore-not-existing</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>ignore</span> <span class="z-string z-quoted z-single z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&#39;</span>../*<span class="z-punctuation z-definition z-string z-end z-shell">&#39;</span></span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>ignore</span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>/*<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>o</span> target/coverage/tests.lcov</span> </span></code></pre> <p>Then, it’s up to us what to do with the data:</p> <ul> <li>Tell <strong>our IDE</strong> to find the <code>lcov</code> files for inline coverage highlights</li> <li>Upload the <code>lcov</code> files <strong>to code coverage providers</strong> such as <a href="https://codecov.io">Codecov</a></li> <li>Use the <code>lcov</code> data to create <strong>custom reports</strong> or to fail tests</li> </ul> <p>When done, we’ll need to clean up the <code>*.profraw</code> files that were littered alongside our code.</p> <h2 id="automating-with-xtask">Automating with <code>xtask</code></h2> <p><code>cargo-xtask</code> is the current implementation of <strong>cargo tasks</strong>. Think of it as a Rust-based alternative to <code>make</code>, but that it is <strong>pure Rust</strong>, all around -- it's just another project in your workspace, and there are very little strict rules around it other than a few <strong>best practices</strong>. To read more about it <a href="https://github.com/matklad/cargo-xtask">check out the xtask repo</a></p> <p>Here’s how all of this process looks with my current flavor of <code>cargo xtask cover</code> (you can take a look at the full example in my <a href="https://github.com/rusty-ferris-club/rust-starter">rust-starter</a>):</p> <p>Now if you run:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo xtask coverage</span> </span></code></pre> <p><strong>You’ll have</strong> <code>lcov</code> <strong>files waiting for you in</strong> <code>coverage/</code> which you can upload to a coverage provider or use in other tools.</p> <p>And if you run:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo xtask coverage<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>dev</span></span> </span></code></pre> <p>You’ll be offered to <strong>open a report</strong> which you can load in your browser, no other coverage product or provider needed.</p> <h2 id="inline-coverage-with-vscode">Inline coverage with VSCode</h2> <p><img src="https://rng0.io/blog/how-to-do-code-coverage-in-rust/2.jpg" alt="" /></p> <p>If you want to see coverage <strong>inline</strong> with your code, as you type, you can do that with the <a href="https://marketplace.visualstudio.com/items?itemName=ryanluker.vscode-coverage-gutters">coverage gutters</a> extension, which is recommended to use with any language that can produce <code>lcov</code> really, so not only useful for Rust.</p> <p><strong>After you’ve installed it</strong>, you need to let it know where to find <code>lcov</code> files generically in your project.</p> <p>For us, if you use the <code>xtask</code> above for all Rust projects, it will always be in: <code>coverage/tests.lcov</code>.</p> <p>So this is the configuration I have, as an example:</p> <pre data-lang="json" class="language-json z-code"><code class="language-json" data-lang="json"><span class="z-source z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>coverage-gutters.coverageFileNames<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span>: <span class="z-meta z-sequence z-json"><span class="z-punctuation z-section z-sequence z-begin z-json">[</span> </span></span><span class="z-source z-json"><span class="z-meta z-sequence z-json"> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>coverage/tests.lcov<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-sequence z-json"> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>lcov.info<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-sequence z-json"> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>cov.xml<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-sequence z-json"> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>coverage.xml<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-sequence z-json"> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>jacoco.xml<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-sequence z-json"> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>coverage.cobertura.xml<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span> </span></span><span class="z-source z-json"><span class="z-meta z-sequence z-json"> <span class="z-punctuation z-section z-sequence z-end z-json">]</span></span>, </span></code></pre> <p>You can then view your inline coverage with <em>watch</em> or just toggle it:</p> <p><img src="https://rng0.io/blog/how-to-do-code-coverage-in-rust/3.jpg" alt="" /></p> <h2 id="running-in-your-ci">Running in your CI</h2> <p>Next, we want to use <em><strong>the same coverage task</strong></em> in our CI, which is another benefit we get by using <code>xtask</code>. Basically, running our coverage in CI will reduce to:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo xtask coverage</span> </span></code></pre> <p>And finding a way to upload <code>coverage/*.lcov</code> to our coverage provider.</p> <p>In this example, we’ll use <strong>Github Action</strong> as our CI, and <strong>Codecov</strong> as our provider, but it shouldn’t be too different for any other set of vendors.</p> <p>Our <strong>Github workflow</strong> should also include set up for the various coverage tools we need, so first install <code>llvm-tools-preview</code>.</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">name</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">Install toolchain</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">id</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">toolchain</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">uses</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">actions-rs/toolchain@v1</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">with</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">toolchain</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">stable</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">target</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">x86_64-unknown-linux-musl</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">override</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-constant z-language z-boolean z-yaml">true</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">components</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">llvm-tools-preview</span> </span></code></pre> <p>Then, we download <code>grcov</code>, work through the rest of our task, and upload to Codecov:</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">name</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">Download grcov</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">run</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-keyword z-control z-flow z-block-scalar z-literal z-yaml">|</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-block z-yaml"> mkdir -p &quot;${HOME}/.local/bin&quot; </span></span><span class="z-source z-yaml"><span class="z-string z-unquoted z-block z-yaml"> curl -sL https://github.com/mozilla/grcov/releases/download/v0.8.10/grcov-x86_64-unknown-linux-gnu.tar.bz2 | tar jxf - -C &quot;${HOME}/.local/bin&quot; </span></span><span class="z-source z-yaml"><span class="z-string z-unquoted z-block z-yaml"> echo &quot;$HOME/.local/bin&quot; &gt;&gt; $GITHUB_PATH </span></span></code></pre> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">name</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">Run xtask coverage</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">uses</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">actions-rs/cargo@v1</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">with</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">command</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">xtask</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">args</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">coverage</span> </span></code></pre> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">name</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">Upload to codecov.io</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">uses</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">codecov/codecov-action@v3</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">with</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">files</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">coverage/*.lcov</span> </span></code></pre> <p>That’s it, at this point you should have:</p> <ol> <li><strong>Local coverage</strong> tools and task for fast feedback</li> <li><strong>Smooth and seamless</strong> IDE integration</li> <li><strong>CI integration</strong> with a nice code coverage provider for working in a team and automatically covering PRs</li> </ol> <p>You can find the <strong>full workflow</strong> in the <a href="https://github.com/rusty-ferris-club/rust-starter/blob/master/.github/workflows/build.yml#L58">rust-starter</a> project.</p> <p>You can also use <a href="https://github.com/rusty-ferris-club/rust-starter"><strong>the project itself</strong></a> as a starting point to get all of those benefits <strong>before writing a single line of code</strong>.<em>Originally published at</em> <a href="https://rng0.io/blog/how-to-do-code-coverage-in-rust/"><em>https://blog.rng0.io</em></a> <em>.</em></p> Principles of Effective Work Wed, 06 Apr 2022 00:00:00 +0000 Unknown https://rng0.io/blog/principles-of-effective-work/ https://rng0.io/blog/principles-of-effective-work/ <p><img src="https://rng0.io/blog/principles-of-effective-work/1.jpg" alt="" /></p> <p>Courtesy of <a href="https://thenounproject.com/scopio/" title="View Profile">Scopio</a> / Noun Project</p> <p><a href="https://spectralops.io">Spectral</a> is a DevSecOps startup which I founded, that was born directly into COVID times (Spectral has since been acquired by Check Point 🎉).</p> <p>When I founded the company it was not clear to anyone that COVID is there to stay (it was early COVID days), and obviously it is still with us for more than 2 years and counting.</p> <p>What <em>was</em> very obvious is that to build a successful company in such huge uncertainty, and a good chance that we won’t actually <em>meet</em> through building the company, through hiring, or working together, introduces challenges around effectively forming a core culture that is as <a href="https://leanpub.com/effectivework"><em>effective as I wanted it to be</em></a>.</p> <p>Even before COVID, we wanted to be a distributed-first company, but there was no pressure. But <em>choosing</em> to be a distributed company versus being forced to be (by COVID) one is different. As COVID evolved into the world epidemic that it is today, we had to improvise our way into being a full-on distributed company AND form a good and effective culture.</p> <p>And so, on day 1, I set out to form some principles (that evolved) that we worked by to form a cultural backbone for a distributed company, winning over challenges and biases the are unique to working in distributed mode or just empower healthy work ethics.</p> <p>Below, is the <em>original page from our confluence</em> (plus/minus some grammar fixes ;-). The format is a short: quick easy summary of the principle, followed by a few examples or suggestions to inspire what it means.</p> <p>Hopefully, you can find it useful in your company to be more effective every day.</p> <h3 id="principles-at-spectral">Principles at Spectral</h3> <ul> <li>Leadership &amp; Teamwork</li> <li>Trust by Default</li> <li>Explicit is Better than Implicit</li> <li>Go Start to Finish</li> <li>Just Enough Design (Engineering)</li> <li>Humans are Built for Duplex Communication</li> <li>Communicate Transparently and Actionably</li> <li>Communication Channels: More is Less</li> <li>Prefer Building Your Ideas Over Talking About Them</li> <li>Before Building, Explore Reusing</li> <li>Cultivate Long Time Stretches</li> <li>No Hand Waving</li> <li>Big Decisions Need Big Research</li> <li>Only One Person Holds the Ball at One Given Time</li> <li>We’re Meeting Averse</li> <li>Everyone is an Analyst</li> <li>Develop a Peripheral Vision</li> </ul> <h3 id="preamble">Preamble</h3> <p>Every company is created in a different way, and share some kind of explicit or implicit principles. If you are a bank, you have a certain set of principles, if you’re a Fortune 500 tech company you have another set, and if you’re a startup you have a different set.</p> <p>If you have time, get a copy of <a href="https://www.amazon.com/Principles-Life-Work-Ray-Dalio/dp/1501124021" title="https://www.amazon.com/Principles-Life-Work-Ray-Dalio/dp/1501124021">Principles: Life and Work</a>, if not, just <a href="https://en.wikipedia.org/wiki/Principles_%28book%29" title="https://en.wikipedia.org/wiki/Principles_(book)">get a summary</a>.</p> <p>In short: principles are a kind of in-betweens of code of behavior and are a determining factor for company culture.</p> <p>Spectral has its own set of principles, and it’s detailed down below.</p> <h2 id="our-principles">Our Principles</h2> <h2 id="leadership-teamwork">Leadership &amp; Teamwork</h2> <p>This is how we see leadership. Best summarized by this 30min watch:</p> <p>Please do invest the time in watching it. IMHO the lecture is more engaging than the book itself, but you can also read the book if you prefer.</p> <h2 id="trust-by-default">Trust by Default</h2> <p>In some companies trust has to be “earned”, which means when you join, you need to earn your stars and stripes through hard work. We think that it’s wrong, unfair and not scalable. At Spectral, we trust by default — at the moment you join you’re given responsibilities as an equal, and we fully trust you.</p> <blockquote> <p><em>We’re all trying to do our best assume the best intentions from people. see Poe’s law</em> <a href="https://en.wikipedia.org/wiki/Poe%27s_law" title="https://en.wikipedia.org/wiki/Poe%27s_law"><em>https://en.wikipedia.org/wiki/Poe's_law</em></a></p> </blockquote> <blockquote> <p><em>When someone cancels a meeting on you, assume their house was burning and they had to deal with the fire, over a malicious intent.</em></p> </blockquote> <blockquote> <p><em>When you feel misunderstood, or a lack of trust; speak from your heart. Talk about how it feels — it’s OK. When all else fails, you can also raise the issue with your team leader.</em></p> </blockquote> <h2 id="explicit-is-better-than-implicit">Explicit is Better than Implicit</h2> <p>Right off the bat: most companies have <em>implicit</em> principles. If you join you need to spend a year or two to “get” the company DNA.</p> <p>By listing our principles explicitly, we associate ourselves with a different breed of companies, who prefer working in the open and sharing.</p> <blockquote> <p><em>Explicit principles are better than implicit ones.</em></p> </blockquote> <blockquote> <p><em>Setting expectations is better than hoping for them</em></p> </blockquote> <blockquote> <p><em>Asking again is better than being shy</em></p> </blockquote> <blockquote> <p><em>Don’t try to shorten your words in written conversation — initials/acronyms are rarely obvious in distributed context. Say the thing, unless it’s really obvious, which is rare to a general audience (which is the entire company). Prefer “Minimum Viable Product” over “MVP”.</em></p> </blockquote> <h2 id="go-start-to-finish">Go Start to Finish</h2> <p>We believe a work half done is worse than a work that never started. When handing off a task, project, or anything really to a colleague, make sure it’s nicely wrapped, and not half baked. By handing over half-baked work, you’re forcing cognitive load, undo and redo of your work.</p> <blockquote> <p><em>Decide to give it an extra push is better than leaving something open</em></p> </blockquote> <blockquote> <p><em>If you can’t give it an extra push, turn it into a POC</em></p> </blockquote> <blockquote> <p><em>If you can’t turn it into a POC, give it some nice documentation</em></p> </blockquote> <blockquote> <p><em>If you can’t do that, write an email/message with ‘danger: half baked’ and arm your colleague with enough information to succeed.</em></p> </blockquote> <h2 id="just-enough-design-engineering">Just Enough Design (Engineering)</h2> <p>Strive to have just enough design. In other words, when you’re designing things — leave wiggle room to revert your decisions without too much collateral damage.</p> <p>You’ll always be learning in your first iteration and we want to get to the next iterations with knowledge we didn’t have in the first place. Big design upfront is a practice that takes into account the following: “There’s nothing to learn here, we know everything that’s there to know, let’s go for it.”. In some industries this is correct. But in most, it’s almost absurd to know everything there is to know, off the bat, right?</p> <blockquote> <p><em>Design just enough for the feature, but then a little bit more.</em></p> </blockquote> <blockquote> <p><em>No big design upfront</em></p> </blockquote> <blockquote> <p><em>Design just enough, ask someone for feedback, and then give it an extra push</em></p> </blockquote> <h2 id="humans-are-built-for-duplex-communication">Humans are Built for Duplex Communication</h2> <p>If you want to do critical thinking and discussion — do it in a pair. If you have a group that needs to be involved, involve it after you have some kind of solid idea. We’re built for a dialogue — we have one set of ears and one mouth; and so — we’re most efficient transferring and receiving information in “duplex mode”, if you’re into engineering analogies.</p> <blockquote> <p><em>If you have a deep design to do, make sure you have exactly two people in the meeting, not more.</em></p> </blockquote> <blockquote> <p><em>It’s OK to say — we don’t know, need to investigate, we’ll split to pairs and come back</em></p> </blockquote> <h2 id="communicate-transparently-and-actionably">Communicate Transparently and Actionably</h2> <p>We think total, fanatic, and uber-transparency is bad, because many times chasing perfect transparency beats the purpose. But we do think that transparency is crucial, and that we open up everything we can, down to common sense. Here’s an example — after any session that feels productive you should ask yourself: “What can I sum up and send as a note to my coworkers, so that they get up to speed with my session?”</p> <blockquote> <p><em>Not every meeting has to have meeting notes. Important ones should.</em></p> </blockquote> <blockquote> <p><em>Every meeting should have something that’s actionable, document that, share .</em></p> </blockquote> <blockquote> <p><em>Beware of meetings about meetings, followups on followups. This means you’re either (a) haven’t been preparing (b) havn’t been vigilant enough to extract actions from the meeting</em></p> </blockquote> <h2 id="communication-channels-more-is-less">Communication Channels: More is Less</h2> <p>Avoid communication channel fatigue — more is less. It’s easy to send your messages across: email, whatsapp, slack all at once in a blitz. It’s very hard for the receiving party to understand where to get what. Have only a few (one?) communication channels, and a clear map of what they’re good for.</p> <blockquote> <p><em>Slack should be used for work, Whatsapp for off-work stuff. Don’t mix-and-match.</em></p> </blockquote> <blockquote> <p><em>Avoid creating a channel for every topic. Channels should be team-themed.</em></p> </blockquote> <blockquote> <p><em>It should be clear what every channel deals with, and where to get updated with what kind of information. It’s OK to kill unclear channels — in fact, developing an obsession to kill such channels is healthy.</em></p> </blockquote> <h2 id="prefer-building-your-ideas-over-talking-about-them">Prefer Building Your Ideas Over Talking About Them</h2> <p>Building is cheap, talk is expensive. Wait what? isn’t the saying “Talk is cheap..”?. In our case, and especially in a distributed setting, talking in a synchronous way (AKA meetings) is forcing another party into participating right here, right now. You’re expensing your time as well as the other party’s.</p> <blockquote> <p><em>Have an idea about a way to implement a data structure? Instead of talking about how to build it — make a branch, build a strawman implementation, and give someone a PR.</em></p> </blockquote> <blockquote> <p><em>When building is actually expensive, fake it. Build a mock. Draw on a piece of paper.</em></p> </blockquote> <blockquote> <p><em>If you have to discuss ideas (we all have to), better make it over some kind of asset: code, drawing, diagram, document.</em></p> </blockquote> <h2 id="before-building-explore-reusing">Before Building, Explore Reusing</h2> <p>You don’t have to build everything. If there’s something that you can reuse, do that instead of building. This is a different version of “don’t reinvent the wheel” strongly geared towards startup productivity. But do it wisely, so that what you’re building doesn’t become a ball of mud.</p> <p>The time when you build instead of buy will come. And you’ll know when it is time. The other way around — isn’t so clear.</p> <blockquote> <p><em>Get a dashboard theme from themeforest, over building a complete UI dashboard from scratch. Start there.</em></p> </blockquote> <blockquote> <p><em>Pay a professional to go over your blogpost for English grammar instead of spending days sending it to friends, family, colleagues just for the purpose of correcting English.</em></p> </blockquote> <h2 id="cultivate-long-time-stretches">Cultivate Long Time Stretches</h2> <p>Be mindful of Paul Graham’s <a href="http://www.paulgraham.com/makersschedule.html" title="http://www.paulgraham.com/makersschedule.html">Maker’s schedule, Manager’s schedule</a>. Whether you are a maker or manager, you can help at the very least — in not “vandalizing” other people’s schedules.</p> <blockquote> <p><em>When you set a meeting, try to set it at the edges of the day. If you see a clear stretch of time for someone in the “maker schedule”, think twice before taking it.</em></p> </blockquote> <blockquote> <p><em>Startup founders are both in the Maker and Manager schedule. Yeap. If you’re a founder, get used to the idea of impossibilities that you have to beat. If you’re working with founders, knowing this is useful.</em></p> </blockquote> <h2 id="no-hand-waving">No Hand Waving</h2> <p>We see <a href="https://en.wikipedia.org/wiki/Hand-waving" title="https://en.wikipedia.org/wiki/Hand-waving">hand waving</a> as a bias that we fall into when we simply don’t have enough data. Hand waving always comes in disguise — like deep and informed knowledge, but most times it’s just a simple guess. Convince with data; not to say we all have to be analysts, so — convince with <em>some</em> data.</p> <blockquote> <p><em>Ask bias-correcting questions for both organizational and factual fallacies — “</em> who <em>told you?” “can you prove it with data?” “can you give a simple example?” “</em> who <em>wrote it?” “is there something you can show?”</em></p> </blockquote> <blockquote> <p><em>Investing half an hour with an experiment yielding data that supports your argument is shorter than the accumulating amount of time of “small argumentative wins” to classically convince people.</em></p> </blockquote> <blockquote> <p><em>Some times data just doesn’t exist. Or it exists, but it sits beyond a wall that takes an unreasonable amount of time and resources to climb. In that case — make an educated guess, write down your thesis and risks, and go for it. Decide right there and then and take the risk. Make sure to check back later and see where or if you went wrong to learn from it. Some times not deciding is worse than being stuck in a cycle of</em> thinking about <em>taking the right decisions.</em></p> </blockquote> <h2 id="big-decisions-need-big-research">Big Decisions Need Big Research</h2> <p>But many small decisions deceptively appear as big decisions. So you need to first (a) make sure a big decision is really a big one (b) if it is so, invest the time in a proper decision making process.</p> <blockquote> <p><em>Weigh and document pro/cons to get a realistic bearing that’s bias-free.</em></p> </blockquote> <blockquote> <p><em>Document down your context. Your reality right now is different than the one that will exist in a year from now, when your decision will bear fruit.</em></p> </blockquote> <blockquote> <p><em>Rubber-duck your ideas. Project your decision making process to your team (but present it with structure, no “free-form brainstorming”). Get feedback. Think. And get feedback again.</em></p> </blockquote> <h2 id="only-one-person-holds-the-ball-at-one-given-time">Only One Person Holds the Ball at One Given Time</h2> <p>But it’s OK to pass. If two people struggle to hold the ball together, it’ll drop. Hey it makes sense in basketball and also in tech. We all want to get the job done, and sometimes a communication breakdown may create a reality where two people are getting the job done in parallel.</p> <blockquote> <p><em>Make sure each JIRA ticket has a reporter, and then and owner</em></p> </blockquote> <blockquote> <p><em>Every task has a task owner. This is not necessarily the person doing the job, but the one responsible for pushing it forward. Many times this person would be both, but it’s not a must. Call it what you want: “feature owner”, “epic master”, etc.</em></p> </blockquote> <blockquote> <p><em>If you’re handling something (probably something big enough), radiate it to your team. Sure; you have mechanisms such as weeklies, DSMs, but there’s no harm in over-sharing that you’re owning something that you think is important to share.</em></p> </blockquote> <blockquote> <p><em>Ask for help.</em></p> </blockquote> <blockquote> <p><em>Feel free to pass the ball if you think someone can do this better/faster than you, or if you think your time is better spent doing something else. But when you do, make sure there’s someone ready to receive your ball. Too many basketball analogies?</em></p> </blockquote> <h2 id="we-re-meeting-averse">We’re Meeting Averse</h2> <p>A lot have been said about meetings in corporates; <a href="https://mgrush.com/blog/meeting-problems/" title="https://mgrush.com/blog/meeting-problems/">too much</a> in fact. We take a position of — if you follow the principles listed here — then a meeting should look like this (this dynamic changes as the company grows substantially, but we’ll worry about this later — we’re still small enough):</p> <blockquote> <p><em>It’s common between two people; it’s rare between more. In that case it’s common for it to even be a phone call.</em></p> </blockquote> <blockquote> <p><em>Oh, and before all that, if you could pay a few bucks to buy instead of build what’s ever been discussed in the meeting — then you just save a whole lot of time not just in building but also in meeting.</em></p> </blockquote> <h2 id="everyone-is-an-analyst">Everyone is an Analyst</h2> <p>We’d like to promote a data driven approach, and factual thinking. By expecting everyone to give their analysis and take — we are practicing this muscle as a group. Data driven and factual analysis is better than guesswork and randomness, it allows us to be more precise, and hit targets on the first attempt; when we fail, we also know why. Other people call this “execution”.</p> <blockquote> <p><em>When you share a news article, give a short gist + your take/analysis from it. Why is it important? how does it relate to us? what is immediately actionable from it?</em></p> </blockquote> <blockquote> <p><em>When you look at a certain product flow, experience it yourself. Give your analysis. There’s a place for facts</em> and <em>emotion that the product provokes in you and we want to hear.</em></p> </blockquote> <blockquote> <p><em>Use our competitors’ products, analyze our competitors and keep and updated and informed view of them (we actually do this during onboarding). It’s OK to like their product don’t hate it just because they’re competition — hate soon brings blanket dismissal, and dismissal soon brings blindness to their mistakes, and we do want to learn from their mistakes — it’s free learnings, without the battle scars and hits that</em> they <em>got.</em></p> </blockquote> <h2 id="develop-a-peripheral-vision">Develop a Peripheral Vision</h2> <p>Your goal as an individual within our distributed team is to develop a peripheral vision. This happens almost always in small teams of 2–3 people sitting in the same physical place. People <em>just know</em> what’s going on in the product, business, people, and more. Ask questions. Be proactive.</p> <blockquote> <p><em>Working in silos is bad. Simply because there will be a time where you’re expected to jump out of your silo and join a different one.</em></p> </blockquote> <blockquote> <p><em>If you hand over part of your work, see that the person receiving got it properly. Or, the flip side of this: never hand over something that doesn’t work.</em></p> </blockquote> <blockquote> <p><em>Ask questions. Be proactive. It’s worth stating it again. And if you find that you’re asking too much, there’s probably something that we can do company-wide to make sure this information is always visible and transparent. Let us know.</em></p> </blockquote> <p>Good luck!</p> 🔑 Open sourcing Keyscope! Tue, 05 Oct 2021 00:00:00 +0000 Unknown https://rng0.io/blog/open-sourcing-keyscope/ https://rng0.io/blog/open-sourcing-keyscope/ <p><img src="https://rng0.io/blog/open-sourcing-keyscope/1.png" alt="" /></p> <p><a href="https://github.com/SpectralOps/keyscope">Keyscope is an open source</a> one-stop-shop tool built in <a href="https://www.rust-lang.org/">Rust</a>, made for developers and security engineers that need to automatically validate (and rotate, in the future) keys, tokens, passwords that were issued by a wide array of any of the SaaS APIs they use such as: Dropbox, Github, Zendesk, Twilio, and more (currently supporting more than 30 providers).</p> <p>You might also want to take a look at <a href="https://github.com/SpectralOps/service-policy-kit">service-policy-kit</a> which is what powers Keyscope.</p> <h2 id="when-should-you-validate-your-api-keys">When should you validate your API keys?</h2> <p>During different occasions in building software, and throughout the organization, there are a few ways a key needs to be <strong><em>validated</em> to assess an active risk and extract mitigation actions, or <em>invalidated</em></strong> when wanting to remediate a risk to your organization:</p> <ul> <li><strong>When mistakenly exposing a secret or key to the public.</strong> For example, a mistaken commit finding itself to a public repository.</li> <li><strong>When leaking a secret from a customer by mistake.</strong> For example when a customer would send out an issue report on Github Issues, and you’d find one of their keys there.</li> <li><strong>When finding and removing secrets in code</strong>, that shouldn’t have been hardcoded.</li> <li><strong>As a security engineer, when you get a list of keys that should have been invalidated, due to company policies</strong> — but you’re not sure that it’s been actually done, and you have to validate yourself.</li> <li><strong>When a SaaS vendor notifies you of a breach</strong>, and recommends to re-issue all of its keys.</li> </ul> <h2 id="using-keyscope">Using Keyscope</h2> <p>Grab a release from <a href="https://github.com/spectralops/keyscope/releases">releases</a>, or install via Homebrew:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">brew</span></span><span class="z-meta z-function-call z-arguments z-shell"> tap spectralops/tap</span> <span class="z-keyword z-operator z-logical z-and z-shell">&amp;&amp;</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">brew</span></span><span class="z-meta z-function-call z-arguments z-shell"> install keyscope</span> </span></code></pre> <p><img src="https://rng0.io/blog/open-sourcing-keyscope/2.png" alt="" /></p> <p>You can try out <strong>validating a key</strong> for a provider, say, Github (assuming the key is in the <code>GITHUB_TOKEN</code> environment variable):</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> keyscope validate github <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">GITHUB_TOKEN</span></span></span> </span></code></pre> <p>You can see which other providers are supported by running:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> keyscope validate<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>providers</span></span> </span></code></pre> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain"> . </span><span class="z-text z-plain"> : </span><span class="z-text z-plain"> . </span></code></pre> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">twilio:validation </span><span class="z-text z-plain">keyscope validate twilio -p p1 p2 </span></code></pre> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">twitter:validation </span><span class="z-text z-plain">keyscope validate twitter -p p1 </span></code></pre> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">zendesk:validation </span><span class="z-text z-plain">keyscope validate zendesk -p p1 p1 </span></code></pre> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">Total 33 providers available. </span></code></pre> <p>And what parameters are required for a certain provider by running (say, <code>stripe</code>):</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> keyscope validate stripe<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>requirements</span></span> </span></code></pre> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">provider stripe requires: </span><span class="z-text z-plain"> - param: p1 </span><span class="z-text z-plain"> desc: stripe key </span><span class="z-text z-plain">$ </span></code></pre> <p>Finally the general structure of the <code>validate</code> command is:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> keyscope validate PROVIDER<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>p</span> PARAM1 PARAM2 .. PARAM_N</span> </span></code></pre> <h2 id="white-check-mark-validation-key-should-be-active">✅ Validation: key should be active</h2> <p>You can validate a specific provider like so:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> keyscope validate twilio<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>p</span> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">TWILIO_KEY</span></span></span> </span></code></pre> <p>With the general pattern of:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> keyscope validate PROVIDER<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>p</span> PARAM1 PARAM2 ...</span> </span></code></pre> <p>The number of keys/params would change <strong>based on authentication type:</strong></p> <ul> <li><code>Bearer</code> - usually just a single key (token)</li> <li><code>Basic Auth</code> - usually 2 keys: user, password</li> <li><code>OAuth</code> - usually 2 keys: client_id, client_secret</li> <li>And others.</li> </ul> <p><strong>Each provider in Keyscope will tell you what it requires using</strong> <code>requirements</code> <strong>:</strong></p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> keyscope validate twilio<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>requirements</span></span> </span></code></pre> <p>You’ll get a report:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> keyscope<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>verbose</span> validate stripe<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>p</span> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">STRIPE_KEY</span></span></span> </span></code></pre> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">✔ stripe:validation: ok 766ms </span></code></pre> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">Ran 1 interactions with 1 checks in 766ms </span></code></pre> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">Success: 1 </span><span class="z-text z-plain">Failure: 0 </span><span class="z-text z-plain"> Error: 0 </span><span class="z-text z-plain">Skipped: 0 </span></code></pre> <p>And an executable exit code that reflects success/failure.</p> <p>You can use the <code>--verbose</code> flag to see API responses:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> keyscope<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>verbose</span> validate stripe<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>p</span> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">STRIPE_KEY</span></span></span> </span></code></pre> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">✗ stripe:validation: failed 413ms </span><span class="z-text z-plain"> status_code: 200 != 401 Unauthorized </span></code></pre> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">Ran 1 interactions with 1 checks in 413ms </span></code></pre> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">Success: 0 </span><span class="z-text z-plain">Failure: 1 </span><span class="z-text z-plain"> Error: 0 </span><span class="z-text z-plain">Skipped: 0 </span></code></pre> <p>In this case the exit code is <code>1</code> (failure).</p> <h2 id="x-validation-key-should-be-inactive">❌ Validation: key should be inactive</h2> <p>When you are validating keys that are supposed to be inactive, you can use the <code>flip</code> flag. In this mode, a failed API access is a good thing, and the exit code will reflect that.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> keyscope<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>flip</span> validate stripe<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>p</span> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">STRIPE_KEY</span></span></span> </span></code></pre> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">✔ stripe:validation: ok 766ms </span></code></pre> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">Ran 1 interactions with 1 checks in 766ms </span></code></pre> <p>In this case, the key is active — <em>which is bad for us</em>. Using <code>--flip</code>, the exit code will be <code>1</code> (failure).</p> <h2 id="cake-adding-your-own-providers-dropbox">🍰 Adding your own providers: Dropbox</h2> <p>As a simple tutorial here’s how to add Dropbox by yourself.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> keyscope<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>f</span> providers.yaml validate dropbox<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>p</span> MY_KEY</span> </span></code></pre> <p>To validate if a dropbox API key works, we first need to learn about the canonical way to authenticate against that API.</p> <p><strong>First stop, API docs:</strong></p> <ul> <li>Dropbox has an <a href="https://dropbox.github.io/dropbox-api-v2-explorer">API Explorer</a> which is super useful</li> </ul> <p><strong>Next stop, we want to find an API call that is a representative for:</strong></p> <ul> <li>Has to be authenticated</li> <li>Has to indicate that when accessed successfully with our candidate key, the key has some authoritative value. Which means, that if exposed, contains significant risk.</li> </ul> <p>For this example, getting our current account sounds like something that only when we identify who we are — we’re able to do.</p> <p><strong>We’ll select</strong> <a href="https://www.dropbox.com/developers/documentation/http/documentation#users-get_current_account"><strong>get_current_account</strong></a> <strong>.</strong></p> <p>Let’s start forming our interaction. First the needed skeleton: containing the name of the provider ( <code>dropbox</code>), its ID and description below, as well as parameters required and their name and description:</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">dropbox</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">validation</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">request</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">id</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-quoted z-double z-yaml"><span class="z-punctuation z-definition z-string z-begin z-yaml">&quot;</span>dropbox:validation<span class="z-punctuation z-definition z-string z-end z-yaml">&quot;</span></span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">desc</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-quoted z-double z-yaml"><span class="z-punctuation z-definition z-string z-begin z-yaml">&quot;</span>dropbox: valid API credentials<span class="z-punctuation z-definition z-string z-end z-yaml">&quot;</span></span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">params</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">name</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">dropbox_1</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">desc</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">dropbox token</span> </span></code></pre> <p>We keep the name of the parameter with a special convention that helps when feeding keyscope automatically:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">PROVIDER_N </span><span class="z-text z-plain">Where &#39;N&#39; starts in 1 e.g.: </span><span class="z-text z-plain">dropbox_1 </span><span class="z-text z-plain">dropbox_2 </span><span class="z-text z-plain">aws_1 </span><span class="z-text z-plain">... </span></code></pre> <p>Then, details about actually making an HTTP call, as required by Dropbox <strong>(Bearer token authentication).</strong></p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">uri</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">https://api.dropboxapi.com/2/users/get_current_account</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">method</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">post</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">headers</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">Authorization</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">Bearer {{dropbox_1}}</span> </span></code></pre> <p>Note that per standard, all HTTP header fields are actually <em>arrays</em>. It’s OK to always make an array of size <em>one</em> if you only have one value (most common case).</p> <p><strong>We also see <em>variable interpolation</em> here.</strong> Where <code>{{dropbox_1}}</code> will get replaced by keyscope in time before making the actual call.</p> <p><strong>Finally, we want to make sure we answer the question:</strong></p> <ul> <li>What does it mean to have a successful call?</li> </ul> <p>In our case, the Dropbox API call returns <code>HTTP OK</code> on success, which means a <code>200</code> status code.</p> <p><strong>And the final, complete result is this:</strong></p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">dropbox</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">validation</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">request</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">id</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-quoted z-double z-yaml"><span class="z-punctuation z-definition z-string z-begin z-yaml">&quot;</span>dropbox:validation<span class="z-punctuation z-definition z-string z-end z-yaml">&quot;</span></span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">desc</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-quoted z-double z-yaml"><span class="z-punctuation z-definition z-string z-begin z-yaml">&quot;</span>dropbox: valid API credentials<span class="z-punctuation z-definition z-string z-end z-yaml">&quot;</span></span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">params</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">name</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">dropbox_1</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">desc</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">dropbox token</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">uri</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">https://api.dropboxapi.com/2/users/get_current_account</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">method</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">post</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">headers</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">Authorization</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">Bearer {{dropbox_1}}</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">response</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">status_code</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-quoted z-double z-yaml"><span class="z-punctuation z-definition z-string z-begin z-yaml">&quot;</span>200<span class="z-punctuation z-definition z-string z-end z-yaml">&quot;</span></span> </span></code></pre> <p>Meanwhile, you can drop this provider in your own <code>providers.yaml</code> file and run keyscope:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> keyscope<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>f</span> providers.yaml validate dropbox<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>p</span> MY_KEY</span> </span></code></pre> <p>Now you can keep this in your private <code>providers.yaml</code> file <strong>or contribute it back to keyscope</strong> if you think other people might enjoy using it - we're happy to accept pull requests.If you’d like to contribute to <a href="https://github.com/SpectralOps/keyscope">Keyscope</a> or <a href="https://github.com/SpectralOps/service-policy-kit">service-policy-kit</a> — you’re more than welcome. Please feel free to submit PR or just ask around and we’ll be there for explaining how things work.</p> Stopping Supply Chain Attacks With Preflight — Spectral Mon, 31 May 2021 00:00:00 +0000 Unknown https://rng0.io/blog/stopping-supply-chain-attacks-with-preflightspectral/ https://rng0.io/blog/stopping-supply-chain-attacks-with-preflightspectral/ <p><img src="https://rng0.io/blog/stopping-supply-chain-attacks-with-preflightspectral/1.png" alt="" /></p> <p><em>TL;DR: we’ve just released Preflight, a CI tool against supply chain attacks that stops attacks like the recent Codecov hack:</em> <a href="https://github.com/spectralops/preflight"><em>https://github.com/spectralops/preflight</em></a>It’s been a few weeks since the <a href="https://spectralops.io/blog/credentials-risk-supply-chain-lessons-from-the-codecov-breach/">CodeCov</a> hack sent ripples across the software development industry. One of these ripples was an increased awareness of supply chain attacks and the risks they present. That said, it’s easy to see that this awareness is not yet implemented as a best practice in the field.</p> <p><img src="https://rng0.io/blog/stopping-supply-chain-attacks-with-preflightspectral/2.gif" alt="" /></p> <p>Here are a few examples:</p> <p>These are all different forms of remotely running shell scripts and blindly executing them. Believe it or not, these examples are live right now in each of these projects, weeks after <a href="https://about.codecov.io/security-update/">the CodeCov breach</a> was discovered and mitigated. An attack in which, as you may remember, this very script was altered to contain malicious code.</p> <p>But <strong>the CodeCov breach is only a symptom of a bigger, more complex issue — supply chain attacks</strong>.</p> <h2 id="codecov-and-supply-chain-attacks">CodeCov and Supply Chain Attacks</h2> <p>The above examples of remote CodeCov bash execution may vanish over time. However, <strong>other instances of remote shell script execution are prevalent even across projects and repositories on GitHub.</strong></p> <p>These create an attack vector for attackers and malefactors. There are numerous types of attacks in this space, with different motivations behind them.</p> <ul> <li> <p><strong>Malicious scripts</strong></p> <p>Since an install script, or a “quick tool” script, is by itself runnable software? The script can be replaced with any other piece of code that performs an attacker’s goal</p> </li> <li> <p><strong>Swapping software</strong></p> <p>A script that we pull can still feel like it behaves nicely. But then, it would let us install a different software package than what we originally thought it would.</p> </li> <li> <p><strong>Tricking into old updates</strong></p> <p>To avoid being detected for giving you a tainted binary? An attacker may simply push an older version of the same software you’re installing, assuming that this older version contains a famous exploit they can use.</p> </li> <li> <p><strong>Tricking into future updates or “update jacking”</strong></p> <p>An attacker might want to point the script at their own update package. Which, in terms of server, is well into the future. So rollback attempts would fail. For the same goal, an attacker can replace a specific update in the past, present, or future (as long as they can cause an updater to trigger).</p> </li> </ul> <p>At the time of writing, these were all live:</p> <p><img src="https://rng0.io/blog/stopping-supply-chain-attacks-with-preflightspectral/3.png" alt="" /></p> <p>Granted, we haven’t discovered anything new here — just a best practice still waiting to become a best practice. <strong>But what would that best practice be?</strong></p> <p>According to CodeCov and their own analysis of their breach:</p> <p>The answer seems quite simple — <strong>validate and verify remotely executed bash scripts</strong>.</p> <h2 id="introducing-preflight">Introducing: Preflight</h2> <p><img src="https://rng0.io/blog/stopping-supply-chain-attacks-with-preflightspectral/4.gif" alt="" /></p> <p><strong>We’ve built</strong> <a href="https://github.com/spectralops/preflight/releases"><strong>Preflight</strong></a> <strong>as a tool to implement that advice into a best practice that is easy to use and integrate into your DevOps workflow</strong>. With a few more features and the icing on top, preflight is now a <em>platform</em> for checking, validating, and running 3rd party components aimed to help you block supply chain attacks.</p> <p><strong>preflight helps you verify scripts and executables and mitigate supply chain attacks.</strong></p> <h2 id="how-does-preflight-work">How Does Preflight Work?</h2> <p>What happens when you execute a “ <code>curl | sh</code> " karate-chop? Here's one breakdown of such a chain in very general terms, highlighting the major links in the chain:</p> <p>When we pull a shell script from a website, we’re trusting the tool we use (curl), and the OS for SSL (CA chain), and the domain (and by derivative our DNS services), and the human operating the domain, and the code hosted on the domain, and the human operating the hosting and code. That’s quite a few links in a chain.</p> <p><strong>With Preflight we’re moving the focus of the chain of trust to a single place: the end of the chain. A signature of the executable which you want to run. The signature is produced and maintained by you, and only you.</strong></p> <p>Although there are many ways to verify trust throughout the entire chain? These would be of more value to software producers (think software factory) to enable forensics, traceability, and more.</p> <p>For our use case, and as depicted in the original Codecov hack? We can get all the benefits of securing our <em>usage</em> of the 3rd party software as <em>consumers.</em> Which is simpler than considering the entire chain of supply on the producer side.</p> <h2 id="getting-preflight">Getting Preflight</h2> <p>First of all, it’s the chicken and the egg. How do you pull a legit <code>preflight </code> binary from us without verifying it with <code>preflight</code>? Having that <code>preflight </code> solves this exact problem?</p> <p><strong>The best way is to grab the source, compile it yourself, and use your own binary which you put in a place <em>that you trust</em>.</strong> You will usually have several options to do that safely:</p> <ul> <li>Put it on your own S3 bucket</li> <li>Drop it on your own <a href="https://jfrog.com/artifactory/">Artifactory</a> or similar</li> <li>Push it directly into your repos (it should be as small as 4mb and seldom change so Git should work nicely with it)</li> </ul> <p><strong>If you want to just get started quickly on your workstation, you can</strong> <a href="https://github.com/spectralops/preflight/releases"><strong>download a release</strong></a> <strong>or install</strong> <code>preflight </code> <strong>with homebrew</strong>:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> brew tap spectralops/tap</span> <span class="z-keyword z-operator z-logical z-and z-shell">&amp;&amp;</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">brew</span></span><span class="z-meta z-function-call z-arguments z-shell"> install preflight</span> </span></code></pre> <h2 id="flying-with-preflight">Flying With Preflight</h2> <p>The curl | sh workflow is packaged as a tool, verified with our own signature that we can create with Preflight. In addition, Preflight can verify any runnable against several of providers such as VirusTotal and others to detect malware automatically</p> <h2 id="so-how-does-it-look-in-action">So how does it look in action?</h2> <p><strong>Someone changed the script or binary you’re running. Abort!</strong></p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> curl<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>L</span> https://XXX</span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">preflight</span></span><span class="z-meta z-function-call z-arguments z-shell"> run sha256=1ce...2244a6e86 ⌛️ Preflight starting ❌ Preflight failed: Digest does not match. Expected: <span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>...<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> Actual: <span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>...<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> Information: It is recommended to inspect the modified file contents.</span> </span></code></pre> <p><strong>A hash is verified, but it is actually vulnerable. Abort!</strong></p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> curl<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>L</span> https://XXX</span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">preflight</span></span><span class="z-meta z-function-call z-arguments z-shell"> run sha256=1ce...2244a6e86 ⌛️ Preflight starting using file lookup: malshare.current.sha256.txt ❌ Preflight failed: Digest matches but marked as vulnerable. Digest matches but marked as vulnerable. Information: Vulnerability: Hash was found in a vulnerable digest list More: malshare.current.sha256.txt</span> </span></code></pre> <p>All ok, let’s fly.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> curl<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>L</span> https://XXX</span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">preflight</span></span><span class="z-meta z-function-call z-arguments z-shell"> run sha256=1ce...2244a6e86 ⌛️ Preflight starting ✅ Preflight verified ... actual script output ...</span> </span></code></pre> <h2 id="codecov-revisited">Codecov Revisited</h2> <p><strong>A perfect example of Preflight in action would be CodeCov</strong>.</p> <p>First, let’s create a hash. Before creating it, be sure to review the script manually and see that it’s not doing anything funny:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> curl<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>s</span> https://codecov.io/bash</span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">./preflight</span></span><span class="z-meta z-function-call z-arguments z-shell"> create sha256=d6aa3207c4908d123bd8af62ec0538e3f2b9f257c3de62fad4e29cd3b59b41d9</span> </span></code></pre> <p>Now, we’re going to take</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">sha256=d6aa3207c4908d123bd8af62ec0538e3f2b9f257c3de62fad4e29cd3b59b41d9 </span></code></pre> <p>And use this to secure our pulls from CodeCov. In this case, <code>preflight </code> is checked safely into your repo under <code> ci/preflight</code>.</p> <p><strong>BEFORE (insecure):</strong></p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">steps</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">curl -s https://codecov.io/bash | sh</span> </span></code></pre> <p><strong>AFTER (safe, yay!):</strong></p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">steps</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">curl -s https://codecov.io/bash | ./ci/preflight run sha256=d6aa3207c4908d123bd8af62ec0538e3f2b9f257c3de62fad4e29cd3b59b41d9</span> </span></code></pre> <h2 id="dealing-with-change">Dealing With Change</h2> <p><strong>When updating an old binary or script to a new updated version, there will be at least two (2) valid digests “live”</strong>. Just replacing the single digest used will fail for the older runnable which may still be running somewhere.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> preflight <span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>hash list</span><span class="z-keyword z-operator z-logical z-pipe z-shell">|</span><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">https://url/to/hash-list</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> </span></span></code></pre> <p>To support updates and rolling/auto-updates of scripts and binaries we need to validate against <code>&lt;old hash&gt; </code> + <code>&lt;new hash&gt;</code> at all times, until everyone upgrades to the new script. Preflight validates against a <code> list of hashes</code>. Or better: <strong>give it a <em>live</em> URL of valid hashes and it will validate against it</strong>.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">curl</span></span><span class="z-meta z-function-call z-arguments z-shell"> ..</span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">./ci/preflight</span></span><span class="z-meta z-function-call z-arguments z-shell"> run sha256=d6aa3207c4908d123bd8af62ec0538e3f2b9f257c3de62fad4e29cd3b59b41d9,sha256=<span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>new hash<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span>,...</span> </span></code></pre> <p>Or to a live URL:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">curl</span></span><span class="z-meta z-function-call z-arguments z-shell"> ..</span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">./ci/preflight</span></span><span class="z-meta z-function-call z-arguments z-shell"> run https://dl.example.com/hashes.txt</span> </span></code></pre> <p>Use this when:</p> <ul> <li>You employ multiple digests verbatim and when your runnables change often (but not too often)</li> <li>You use a URL when your runnables change often. Remember to follow the chain of trust. This will now mean that:</li> <li>Your hash list URL is now a source of trust</li> <li>Visually: we’re swapping the chain of trust like so <code>curl &lt;foreign trust&gt; | ./ci/preflight &lt;own trust&gt;</code></li> </ul> <h2 id="checking-scripts-and-binaries">Checking Scripts And Binaries</h2> <p><strong>If you want to use Preflight as another tool for improving your Unix pipes chops, you can with the “check” command</strong>:</p> <h3 id="piping">Piping:</h3> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> curl<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>s</span> https://example.com/some-script</span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">preflight</span></span><span class="z-meta z-function-call z-arguments z-shell"> check sha256=d6aa3207c<span class="z-constant z-numeric z-integer z-decimal z-file-descriptor z-shell">4</span><span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>...<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span>b4</span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sh</span></span> </span></code></pre> <p>Note that <code>preflight check</code> is built in a way that allows you to continue to pipe to the next process:</p> <ul> <li>If a check passes, the checked script or binary content will be dumped to <code>STDOUT</code></li> <li>If a check fails, you’ll get an <code>exit(1)</code>, and an error message</li> </ul> <h3 id="executables">Executables:</h3> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> preflight check sha256=d6aa3207c<span class="z-constant z-numeric z-integer z-decimal z-file-descriptor z-shell">4</span><span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>...<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span>b4 ./my-script.sh</span> </span></code></pre> <p>In this case:</p> <ul> <li>If a check passes, you’ll get an <code>exit(0) </code> and no output (so you can compose with other tooling)</li> <li>If a check fails, you’ll get an <code> exit(1)</code> and an error message</li> </ul> <h2 id="creating-new-hashes">Creating New Hashes</h2> <p><strong>You can easily create new hashes with preflight</strong>. The default is a <code>SHA256 </code> hash, but you could also create a <code>sha256</code>, <code>sha1</code>, and <code>md5 </code> hash.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> preflight create test.sh sha256=fe6d02cf15642ff8d5f61cad6d636a62fd46a5e5a49c06733fece838f5fa9d85</span> </span></code></pre> <p>Though not recommended, you can create other kinds (weaker kinds) of hashes for legacy/compatibility reasons:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> preflight create test.sh<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>digest</span> md5 md5=cb62874fea06458b2b0cabf2322c9d55</span> </span></code></pre> <h2 id="malware-lookup">Malware lookup</h2> <p><code>preflight </code> <strong>comes with lookup providers</strong>. This is optional and you can enable them by using environment variables:</p> <h3 id="file-lookup">File Lookup</h3> <p><strong>You can download a daily list of malware signatures from malshare.com or any equivalent service</strong>. Here is a <a href="https://www.malshare.com/daily/malshare.current.sha256.txt">direct link to such a list</a>.</p> <p>Then:</p> <p>With this configured preflight will search for all digest types in this file before approving.</p> <p>Here is a full example for your CI, combining <code>preflight </code> with Malshare:</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">env</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">PF_FILE_LOOKUP</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">malshare.current.sha256.txt steps</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">wget https://www.malshare.com/daily/malshare.current.sha256.txt - curl https://... | preflight &lt;sha&gt;</span> </span></code></pre> <p>Result:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> PF_FILE_LOOKUP=malshare.current.sha256.txt preflight run fe6d02cf15642ff8d5f61cad6d636a62fd46a5e5a49c06733fece838f5fa9d85 test.sh ⌛️ Preflight starting using file lookup: malshare.current.sha256.txt ❌ Preflight failed: Digest matches but marked as vulnerable. Information: Vulnerability: Hash was found in a vulnerable digest list More: malshare.current.sha256.txt</span> </span></code></pre> <h3 id="virustotal-lookup">VirusTotal Lookup</h3> <p>You can use the virus total community API access to lookup your hashes.</p> <p><strong>With this configured,</strong> <code>preflight </code> <strong>will automatically create the VirusTotal lookup provider and validate the digest with it</strong>.</p> <p>Here is a full example for your CI, combining preflight with VirusTotal:</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">env</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">PF_VT_TOKEN</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-meta z-flow-mapping z-yaml"><span class="z-punctuation z-definition z-mapping z-begin z-yaml">{</span><span class="z-meta z-flow-mapping z-yaml"><span class="z-punctuation z-definition z-mapping z-begin z-yaml">{</span><span class="z-string z-unquoted z-plain z-in z-yaml">secrets.PF_VT_TOKEN</span><span class="z-punctuation z-definition z-mapping z-end z-yaml">}</span></span><span class="z-punctuation z-definition z-mapping z-end z-yaml">}</span></span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">steps</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">curl https://... | preflight &lt;sha&gt;</span> </span></code></pre> <p>Result:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> PF_VT_TOKEN=xxx preflight check e86d4eb1e888bd625389f2e50644be67a6bdbd77ff3bceaaf182d45860b88d80 kx-leecher.exe ⌛️ Preflight starting using VirusTotal ❌ Preflight failed: Digest matches but marked as vulnerable. Information: Vulnerability: VirusTotal stats - malicious: 40, suspicious 0 More: https://www.virustotal.com/gui/file/e86d4eb1e888bd625389f2e50644be67a6bdbd77ff3bceaaf182d45860b88d80/detection</span> </span></code></pre> <h2 id="summary">Summary</h2> <p>While there’s a lot of interest in protecting the use of 3rd party libraries? Frequently — especially in CI systems — we’re actually <em>running</em> 3rd party tools to get a certain job done. This is another chance for attackers to aim at our supply chain. Using Preflight, we can ensure we employ 3rd party tools safely and securely.</p> <p>Preflight is open source, and we’re accepting pull requests so feel free to <a href="https://github.com/spectralops/preflight">visit our repo on Github</a></p> The last mile of sensitive data — solved with Teller Sun, 23 May 2021 00:00:00 +0000 Unknown https://rng0.io/blog/last-mile-of-sensitive-datasolved-with-teller/ https://rng0.io/blog/last-mile-of-sensitive-datasolved-with-teller/ <p><img src="https://rng0.io/blog/last-mile-of-sensitive-datasolved-with-teller/1.png" alt="" /></p> <p><a href="https://github.com/SpectralOps/teller">Teller</a> is an open source secrets management tool for developers built in Go, that helps you manage, protect, and fix problems in your code and security posture when it comes to your vaults and key stores. It also lets you <em>control all of your stores</em> in one place, in a secure way.</p> <p><img src="https://rng0.io/blog/last-mile-of-sensitive-datasolved-with-teller/2.gif" alt="" /></p> <p><strong>Almost any given modern software project these days will contain a set of technologies that offer a developer-productivity story, for traditional cloud stacks and cloud native stacks. One such stack is your typical Kubernetes stack. Which may contain Kubernetes itself, but also Istio, Helm, Docker and various Docker tools, Kustomize, various development and testing tooling such as k3s or minikube and your set of cloud SaaS vendors to interact with such as AWS, GCE, or Mailgun, Mixpanel, and many others.</strong></p> <p>These all are bound to change. Their features, their documentation, their integration points, API surface area, and above all the best practices of how to use all of these in perfect concert securely. One crucial part of gluing everything together — since these all have to integrate together eventually — is authentication, authorization and secrets and access detail.</p> <p>With these challenges comes a painful disorder for the developers needing to build a coherent, single development experience for themselves, which is almost guaranteed to end up with shortcuts, scripts and makeshift tools to enable access and integration: How do you pull a token from mailgun, inject it into your <a href="https://lightrun.com/debugging/debugging-microservices-the-ultimate-guide/">Kube cluster</a>, but also have the sandbox token in your minikube cluster, and perform an integration test for all of those combined? How do you move from minikube to k3s and take all of those specially-crafted scripts that pull secrets from all kinds of places?</p> <h2 id="developers-developers-developers">Developers, developers, developers.</h2> <p><strong>This small chaos that’s created isn’t great, to say the least. It leads to forgotten keys, misplaced tokens, exposure and leak of secrets and sensitive files, and above all — cause individuals to mistakenly override organizational security controls. You can have great security on your production workloads, but a small chaos in the last mile of it: the habits of day to day software development.</strong></p> <p>When software projects are successful, production workloads are healthy, monitored, alerts are in place, <a href="https://spectralops.io/blog/8-tips-effective-code-review-checklist-xls/">fire-drills</a> and processes are well-known and well-rehearsed, and when something fails, you probably know about it. In terms of <a href="https://spectralops.io/blog/6-steps-data-breach-response-plan/">data safety</a> — sensitive data can travel freely within our production clusters back and forth safely.</p> <p>When it comes to the last mile of handling sensitive data and secrets in software, and more specifically in software development <em>processes</em> and our individual development habits — it is often uncharted territory and a developer workstation can become chaotic.</p> <p>To deal with this last-mile problem, as developers, we use solutions and practices such as 12-factor apps, .env and various next-gen developer-friendly software vaults, such as Hashicorp Vault and cloud native developer-friendly key value stores like Consul and Etcd (all of which open source!). Still, often it’s hard to visualize what “productive” looks like with all these, and to create a streamlined, razor-sharp and secure workflow to support developer productivity <em>and also</em> security.</p> <h2 id="never-leave-your-command-line-for-secrets">Never leave your command line for secrets</h2> <p><img src="https://rng0.io/blog/last-mile-of-sensitive-datasolved-with-teller/3.png" alt="" /></p> <p>Secret sprawl is a semi-standard term for the bad practice that happens when your organization stores sensitive information such as secrets, keys, and other types of sensitive data in many different places. In order to make life easy, and lacking any proper solution, developers may place such secrets in code itself — hardcoded, in configuration files, in IaC (infrastructure as code) repos, in their private workstation setup (shell environment, zsh, IDEs) and more.</p> <p>When secret sprawl initially happens — as a function of organization growth — the problem grows exponentially. For a healthy and growing organization, codebases grow, people duplicate, share, or release code internally, which gets shared once again, and the network effect takes care of growing secret sprawl massively inside the organization.</p> <p>Often secret sprawl happens without individuals realizing it, because it’s naturally hard to see the bigger picture, and existing solutions are not doing a good enough job (disclaimer: that’s why we’re building <a href="https://spectralops.io/">Spectral</a>, which solves this problem precisely).</p> <p><img src="https://rng0.io/blog/last-mile-of-sensitive-datasolved-with-teller/4.png" alt="" /></p> <p>Teller removes these risks completely as it acts as the “last mile shipping hub” of secrets on your workstation. And worry not — a given <code>teller.yml</code> file contains no secrets - but instead contains the metadata that is required to fetch these secrets. The secrets are always stored in their best and most secure place to be - the organizational vault or other secure key stores (we support almost all keystores that exist!), Teller makes sure that they are never written to any file or persisted to any storage device.Teller encourages developer environment hygiene, where it will never give a child process being run secrets that the owning shell process may contain — every process is run with a context which is on a need-to-know basis. This has a nice side effect that nullifies the vulnerability where hackers build processes that steal environment variables, often containing tokens and secrets that the developer is using constantly in their development environment.</p> <p>In a micro-services or could-native landscape in the organization, it is very easy to find yourself needing to bring up a couple Go services, as well as a Kafka and a Node service from a few different teams, just to get going. And it’s reasonable to assume that you’re not aware of how these should be set up in terms of their data, context, and secrets. Not only that, it’s quite obvious that, out of security reasons, the originating teams wouldn’t want to share their secrets.</p> <p>Vaults and key stores can be used to provide different teams safe and sandboxed secrets. However, the details of how to get and set these up are baked into READMEs, custom scripts and word-to-mouth, which is far from ideal. Teller can take all of that domain knowledge and pack it into one simple <code>teller.yml</code> file that each team creates. From there on, you just <code>teller run</code> the required services.</p> <p><img src="https://rng0.io/blog/last-mile-of-sensitive-datasolved-with-teller/5.png" alt="" /></p> <p>This can help developers and teams demo their product, tame the complexity and context switch more effectively without the fatigue associated with remembering, instructing, and checklisting these actions.</p> <p>Teller is an open source and free tool that helps developers create a clean and hygienic development environment and helps keep infosec policies and first-class use of existing vaults, key stores and other sensitive data storage facilities in the organization.</p> <p>As a side effect, by merely using Teller, we can block a number of possible attacks on our processes and personal workstations.</p> <p>You can mix and match any kind of vault, key store, or API you can think of to grab keys from and have them ready for your process (everything is safe and secure, in memory).</p> <p>You can <a href="https://github.com/spectralops/teller/releases">get started with Teller</a> right now — take it for a spin or <a href="https://github.com/spectralops/teller">build from source</a>.</p> Why we need developer tools for security and not security tools for developers Mon, 19 Apr 2021 00:00:00 +0000 Unknown https://rng0.io/blog/why-we-need-developer-tools-for-security-and-not-security-tools-for-developers/ https://rng0.io/blog/why-we-need-developer-tools-for-security-and-not-security-tools-for-developers/ <p>The further down the line we discover a software defect — the more it costs to fix and recover from it, whether it’s a bug that leads to visual error like a <a href="https://en.wikipedia.org/wiki/Schiaparelli_EDM">spacecraft crash</a> or a more day-to-day bug, human error, or plainly scale issues, that leads to a <a href="https://www.isitdownrightnow.com/">service being down</a>.</p> <p><img src="https://rng0.io/blog/why-we-need-developer-tools-for-security-and-not-security-tools-for-developers/1.jpg" alt="" /></p> <p><em>Source:</em> <a href="https://www.nasa.gov/image-feature/jpl/pia21132/schiaparelli-impact-site-on-mars-in-color/"><em>Schiaparelli Impact Site on Mars, in Color</em></a></p> <p>In fact, there’s plenty of research and writing about software bugs, their cost, and their <em>real</em> cost. A few examples you can find <a href="https://raygun.com/blog/cost-of-software-errors/">here</a>, <a href="https://www.celerity.com/the-true-cost-of-a-software-bug">here</a> and <a href="https://www.isixsigma.com/industries/software-it/defect-prevention-reducing-costs-and-enhancing-quality/">here</a>.</p> <p>As an industry having to ship software faster and faster each day with limited attention span as a huge, fast moving organism — it has been a slow realization: with the growth of “software eats the world”, software defects — and more critically security defects — are becoming a bigger problem as well.</p> <p><img src="https://rng0.io/blog/why-we-need-developer-tools-for-security-and-not-security-tools-for-developers/2.jpg" alt="" /></p> <p><strong>Security defects have a unique trait built into them — they are always searched for, probed and exploited by an adversary — as long as you have something worth exploiting.</strong> The cost for a security defect and the probability that the worst-case scenario will actually happen, unlike a traditional software defect, could be very real and unfortunately <a href="https://theintercept.com/2020/12/24/solarwinds-hack-power-infrastructure/">be very big</a>.</p> <p>This is why, in a similar effort that was invested to tame software defects, the industry as a whole is looking to tame security defects, or security vulnerabilities at the <em>start of software production</em> and also in the design phases, or for a more popular term — shifting left security.</p> <p>Shifting left security means adoption and <em>adaptation</em> of very much the same processes, practices, tools and capabilities that exist in the parallel world of “traditional” software testing and mitigating defects into security testing and mitigating security vulnerabilities.</p> <p>The big game-changes that we can borrow from the testing world are:</p> <ul> <li>Understanding developers, developer tools, developer workflow and processes (AKA in the testing world: unit testing, e2e tests, stories, and above all the <em>right tools</em> for that)</li> <li>Pushing traditional security processes to the “left” of the software production process (AKA: <a href="https://en.wikipedia.org/wiki/Shift-left_testing">shift left</a>)</li> <li>Putting security tools, awareness, and practices in the hands of developers (AKA: empowering developers)</li> <li>An org structure to support the radical shift to the left, e.g. DevSecOps (AKA in the testing/agile world: squads, guilds, matrix structure)</li> </ul> <h2 id="devsecops-and-shifting-left-gains">DevSecOps and shifting left gains</h2> <p>The premise of DevSecOps and shifting left security is the same: <strong>align security with the rest of the software development lifecycle</strong>.</p> <p>The immediate <em>gains</em> of shifting the discovery of security defects left are:</p> <ul> <li><strong>Detecting and mitigating security issues as early as coding time</strong>. This makes it similar to the gains already made in the software testing domain.</li> <li><strong>Moving from security as an afterthought to</strong> <a href="https://en.wikipedia.org/wiki/Secure_by_design"><strong>security by-design</strong></a>, getting better feedback at the <em>right</em> time of software creation: design and build phase.</li> </ul> <p>In the general sense, we’ve <em>already</em> seen one of the most disruptive changes to our industry already — the move <strong>from Ops to DevOps.</strong></p> <h2 id="developer-tools-for-security">Developer tools for security</h2> <p>For someone looking to build developer-first security tools, or integrating them, or is through the process of selecting and designing their own solution based on those we need to answer the following:</p> <p><img src="https://rng0.io/blog/why-we-need-developer-tools-for-security-and-not-security-tools-for-developers/3.jpg" alt="" /></p> <ul> <li>How to build or evaluate tools for a developer-first approach? Where do we start?</li> <li>Security tools for developers vs developer tools for security? What is the difference?</li> </ul> <p>The criteria or checklist that we need revolves specifically around:</p> <ul> <li><strong>CI</strong> — what’s important to consider in your dev workflows and processes</li> <li><strong>Tooling</strong> — how to evaluate these tools in the context of your dev day to day</li> <li><strong>Feedback cycle</strong> — the other types of feedback cycle opportunity developers might want</li> <li><strong>Processes</strong> — or more specifically threat modeling meets dev, meets security experts meets product.</li> </ul> <h2 id="does-it-work-in-my-ci">Does it work in my CI?</h2> <p>One of the popular criteria for determining the fit of a security tool to be a developer friendly tool is the question: “does it work in my CI?”.</p> <p>Many security tools have CI integration challenges that stem from their design. A funny viewpoint of this, is where <a href="https://twitter.com/abhaybhargav">Abhay Bhargav</a> talked about the phenomenon of <em>security tool narcissism</em> -</p> <p>“Run MY tool, see MY dashboard!”</p> <p>This self-centric and developer-unfriendly approach inverts control over where you want to exercise your guards. Sometimes developers just want to run a tool. Period.</p> <p><img src="https://rng0.io/blog/why-we-need-developer-tools-for-security-and-not-security-tools-for-developers/4.jpg" alt="" /></p> <p>The fact that you need to fiddle with a UI to accomplish a task — destroys the chances of shifting left security. Want to run a scan? Use the UI. Want to see results? Log into the system. Want quick feedback? Log into the system, see status. These ALL are bothersome, and miss the point of building to the zen of development.</p> <p><strong>In addition some of these security tools have inherent <em>security design flaws</em> when running against your securely kept assets</strong>. Whether it’s at the code analysis stage or deployment stage (such as a building a Docker container ready for deployment), a tool that moves your assets (such as your code) off-perimeter, essentially copying everything to a remote system that doesn’t belong to you for the sake of a security analysis — is painfully dangerous: <strong>if you have a security issue (such as exposing sensitive customer data, credit cards, or secrets), it now exists in two physical places, one is yours, and one is stored at a vendor that is not you</strong>.</p> <h3 id="16-essentials-for-building-tools-for-developers">16 Essentials for building tools for developers</h3> <h2 id="1-be-fast">1. Be fast</h2> <p><img src="https://rng0.io/blog/why-we-need-developer-tools-for-security-and-not-security-tools-for-developers/5.jpg" alt="" /></p> <p><strong>Many, if not all CI systems</strong> <a href="https://docs.github.com/en/free-pro-team@latest/github/setting-up-and-managing-billing-and-payments-on-github/about-billing-for-github-actions"><strong>meter by the minute</strong></a> <strong>, and some of them have OS specific multipliers</strong>. Github Actions for example, as of the time of this writing — bills using 10x times multiplier per minute on macOS than on Linux. If a tool takes a long time to run, it may become ineffective and expensive to run.</p> <p>CI is all about creating the primary feedback cycle for a <em>team</em>. <em>The</em> major concern is the speed of your feedback cycle for a given team. CI workflows tend to start lean, and as the team and codebase evolve, they become heavy and long running.</p> <p><strong>When building a CI friendly security tool you think about investing in a technology that can run quickly, and keeping the overall CI times in check.</strong></p> <p>If you went through this process once or twice in your career of watching a fast CI workflow become dead slow — you learned to keep in mind that <strong>every additional tool and process affects CI job time</strong>.</p> <h2 id="2-be-simple">2. Be simple</h2> <p><img src="https://rng0.io/blog/why-we-need-developer-tools-for-security-and-not-security-tools-for-developers/6.jpg" alt="" /></p> <p>Fancy, cryptic, hacker tools exist only in movies. In real life, <strong>when we hit a problem we want to understand it in simple terms and fix it in simple term</strong> s. This means:</p> <ul> <li>Textual output — no fancy reports</li> <li>Point directly to the problem and show me how to fix it (save me from forensics if possible)</li> <li>Fancy reports uploaded as artifacts for extra information</li> <li>Give me the raw data and give me context</li> <li>Don’t take too many keys, and don’t require privileged access if possible</li> <li>Be familiar — the way the tool runs in CI should be similar to the way it runs in development</li> </ul> <p><strong>When running in my CI, don’t copy my assets off premise, don’t phone home without my knowledge, and don’t send my data anywhere else.</strong></p> <h2 id="3-don-t-be-rude">3. Don’t be rude</h2> <p>In addition, don’t break my existing security guards such as:</p> <ul> <li>Dumping CI environment variables by mistake (which might contain sensitive tokens or secrets)</li> <li>Don’t run with privileges you don’t have, weren’t given, or don’t need</li> </ul> <h2 id="4-support-a-test-matrix">4. Support a test matrix</h2> <p><strong>To support the benefits of such a test matrix, if you’re building or adopting a tool for your CI, it should be able to run on many operating systems, and many architectures (did someone</strong> <a href="https://en.wikipedia.org/wiki/Mac_transition_to_Apple_Silicon"><strong>say ARM</strong></a> <strong>?).</strong></p> <p>CI systems support a <a href="https://circleci.com/blog/circleci-matrix-jobs/">test matrix</a> and using it is one of CI’s best values: test the same thing over a matrix of possibilities — quickly. This is exactly the thing a developer <em>cannot</em> do reliably on their machine due to hardware and existing software differences, or wouldn’t want to invest time in (I certainly won’t).</p> <h2 id="5-use-junit-as-a-report-format">5. Use JUnit as a report format</h2> <p>It is almost unbelievable that <a href="https://plugins.jenkins.io/junit/">JUnit test reports</a> are the standard of reporting in all modern CI systems; JUnit being the archetype and workhorse of modern unit testing frameworks, it didn’t disappoint here as well.</p> <p><strong>A tool that behaves well on CI, should be able to output a JUnit format output <em>even though</em> it doesn’t really run unit tests</strong>. The format is so flexible so that every form of testing can be baked into it — including security testing.</p> <h2 id="6-consider-sarif-for-static-analysis">6. Consider SARIF for static analysis</h2> <p>With that said, <strong>for security-specific reporting, we now have the relatively new</strong> <a href="http://docs.oasis-open.org/sarif/sarif/v2.0/csprd01/sarif-v2.0-csprd01.html"><strong>SARIF</strong></a> <strong>format for static analysis results</strong> (stands for: Static Analysis Results Interchange Format). A rather new but powerful format to generate output for, with the primary consumer at the time of this writing — Github.</p> <h2 id="7-have-output-ready-for-deep-dives">7. Have output ready for deep dives</h2> <p>A CI system being out of your reach sometimes (such as a cloud CI service) may require a deeper look into a run session. In that case, <strong>a tool should be able to output deeper reports, more data so that it can be handled offline</strong> — for debugging or understanding the interaction between the CI system, the asset under analysis, and the tool doing the analysis.</p> <p><strong>When you’ve found an issue — an exit code should exist to signal an error state. Examining exit codes is a basic step in composing processes.</strong></p> <p>For example, a CI system that uses containers as a virtualization mechanism, might change its base image without your notice or attention, and suddenly things will start breaking in surprising ways.</p> <h2 id="8-be-composable-aka-unix-philosophy">8. Be composable (AKA UNIX philosophy)</h2> <p>Often you’d want to compose new workflows in your CI. Most CI systems already take into account streaming and exit codes, and so the expectation is that developer and CI friendly tools would behave in an expected way.</p> <h2 id="9-be-easy-to-set-up">9. Be easy to set up</h2> <p><strong>A well-behaving developer tool should have as little dependencies as possible</strong>. Ideally it should be a self-contained binary. Even though you can build container images that wrap everything, and you can set up CI processes with all the required dependencies? These break as dependencies shift, as CI systems evolve, and as your code and requirements change. You don’t want to be investing time in CI set up every time one of those changes.</p> <h2 id="10-auto-update-is-magic-but-it-s-also-surprising">10. Auto-update is magic, but it’s also surprising</h2> <p><a href="https://en.wikipedia.org/wiki/Principle_of_least_astonishment">Don’t surprise me</a>. Principle of least surprise in short here:</p> <p>“People are part of the system. The design should match the user’s experience, expectations, and mental models”.</p> <p>For a developer tool context; <strong>one of the utmost concerns is for a tool to be always accessible and behave exactly the same as was the last time you’ve used it</strong>; given that you use other tools in your toolbox from time to time (but not all the time).</p> <p>An old flag is missing in the new release, or it breaks because a feature was removed. Or even, try to use it and it now decides to auto-update — and you’re staring at your screen now (looking at you, Windows!). This all impairs productivity, which if you take any given developer time-stress these days — is already impaired by the fatigue of the likes of Slack, email, JIRA, information overload, tech complexity, bad project management, and more.</p> <p>PS: This excludes the special and important case of critical updates and security updates — which in any case, I expect to be transparent, non-breaking, non-removing and non-surprising updates.</p> <h2 id="11-no-context-switches">11. No context switches</h2> <p><strong>Don’t force me to use a UI to perform an action on my way to getting my job done</strong>. Put these steps <em>in</em> the tool, in the CLI (e.g. register, login, authenticate, main value, reporting).</p> <p>Imagine if you wanted to publish a package on <a href="http://npmjs.org/">NPM</a>, but to do that you had to leave your terminal, go to your browser, log in, copy some kind of token back, set up a configuration file, and then get back to publishing; just to realize its the wrong account you’ve logged into on your browser because you have your non-work profile there.</p> <p>Nope. You log in through your tool that you’re using and all that happens automatically. Don’t force me to use a UI means — don’t force me to context switch in any case on the way to getting value from your tool.</p> <p>An exception in this case would be an MFA login, which by-design forces you to add another factor and then possibly context switch to your phone or any other form of authenticator that you might have.</p> <h2 id="12-work-on-any-os">12. Work on any OS</h2> <p><strong>Support all operating systems</strong>. I don’t want to set up a VM for you, I don’t want to run in docker. Cross compilation is easy and cheap — you just have to pick the right programming language for it.</p> <p><strong>Have great docs for your tool, <em>in addition</em> to having great docs for your product.</strong></p> <p>Some tools and programming language cargo cult their platform gaps. Whether it’s tools that rose to fame on Windows (SDR, Software Defined Radio tools have their best version only on Windows), or on Linux. As a developer, I’m using all these platforms with the primary one being my mac.</p> <p>It’s not uncommon for my best <em>tools</em> to go with me everywhere; the tools I highly appreciate — I’ll use on my mac, Linux and Windows development workflows.</p> <h2 id="13-have-great-documentation">13. Have great documentation</h2> <p>Developer documentation should have a product manager too (that might be you). The times of man-pages and “dry” docs are gone. Treat your docs like a product, and use <a href="https://jtbd.info/">JTBD</a> to gather <em>requirements</em> for it. If you use this same model, you’ll realize that you want code samples in your docs, and troubleshooting advice in the same paragraph.</p> <p>When you’re authoring docs to <em>display</em> everything you’ve got, as opposed to letting someone <em>perform a given task</em> with the help of the text you’re just writing now — it makes a whole lot of difference.</p> <h2 id="14-the-pit-of-success">14. The pit of success</h2> <p>When I’m not in the mood for reading the docs, <strong>tell me what’s the right next step in the CLI itself</strong>. Also -help should do what you would expect.</p> <p>The <a href="https://blog.codinghorror.com/falling-into-the-pit-of-success/">pit of success</a> principle means, that if i’ve learned to do the first step with your tool, this step should teach me what’s the next reasonable step to perform. <strong>A tool should not only do a certain job, but it also has the role of an educator; such an educator that I get only the right information, on the right time.</strong></p> <h2 id="15-be-responsive">15. Be responsive</h2> <p><img src="https://rng0.io/blog/why-we-need-developer-tools-for-security-and-not-security-tools-for-developers/7.jpg" alt="" /></p> <p>Nielsen has put 3 important limits for usability engineering:</p> <blockquote> <p>0.1 second is about the limit for having the user feel that the system is reacting instantaneously, meaning that no special feedback is necessary except to display the result.</p> </blockquote> <blockquote> <p>1.0 second is about the limit for the user’s flow of thought to stay uninterrupted, even though the user will notice the delay. Normally, no special feedback is necessary during delays of more than 0.1 but less than 1.0 second, but the user does lose the feeling of operating directly on the data.</p> </blockquote> <blockquote> <p>10 seconds is about the limit for keeping the user’s attention focused on the dialogue. For longer delays, users will want to perform other tasks while waiting for the computer to finish, so they should be given feedback indicating when the computer expects to be done. Feedback during the delay is especially important if the response time is likely to be highly variable, since users will then not know what to expect.</p> </blockquote> <p>Out of which, with developer tools, you want to adopt the <a href="https://www.nngroup.com/articles/response-times-3-important-limits/">100ms of UI responsiveness</a> limit. Where it is not only critical in graphical interfaces, but it is also important in the much neglected realm of developers — the terminal.</p> <p>There’s no denying that when you use a fast tool it makes this fuzzy impression on you that you want to keep it. Speaking of fuzzy — <a href="https://github.com/junegunn/fzf">fzf</a> is such a great example (no pun intended), it’s one of the fastest fuzzy searching tools that I keep using as a replacement for the standard history feature in shells.</p> <p>Startup time makes a difference, and run time makes a difference. You can’t use Java for CLI tools; you can’t bring up a mammoth VM for a quick ride (not counting GraalVM and latest Leyden), and then it may not even get to <a href="https://stackoverflow.com/questions/18220837/just-in-time-compilation-when-does-it-actually-take-place-in-java">JITting</a> at all.</p> <h2 id="16-have-that-oh-cool-factor">16. Have that “oh cool” factor</h2> <p><img src="https://rng0.io/blog/why-we-need-developer-tools-for-security-and-not-security-tools-for-developers/8.jpg" alt="" /></p> <ul> <li>Teach me things I don’t know</li> <li>Work well with my existing tools. Install via <a href="https://brew.sh/">Homebrew</a>, integrate with <a href="https://github.com/typicode/husky">Husky</a>, work well with <a href="https://www.vim.org/">Vim</a>.</li> <li>No set up ceremonies — no “what did I do wrong” after install</li> <li>Have somewhere I can complain, and be listening and responsive to my complaints</li> </ul> <h3 id="the-various-speeds-of-feedback-cycles">The various speeds of feedback cycles</h3> <p><img src="https://rng0.io/blog/why-we-need-developer-tools-for-security-and-not-security-tools-for-developers/9.jpg" alt="" /></p> <p>Developers have a few places to get great, fast feedback; choosing which to use depends heavily on the nature of the analysis we want to do, and if the security tool can allow running under that specific context (some security tools can’t run on a dev machine because they’re too heavy).</p> <p>Common to the different feedback cycles and their speed, is the following:</p> <ol> <li><strong>Speed</strong>— feedback must reach me quickly, in order to maintain my productivity, also it must not disrupt others</li> <li><strong>Correctness</strong>— the feedback must be correct. Many times for security tools this means false positive and the handling of it</li> <li><strong>Actionable</strong>— feedback must be actionable. For security tools this should be a clear explanation of the issue and its mitigation as well as education.</li> </ol> <h2 id="ide-integration">IDE integration</h2> <p>This would seem as the fastest feedback cycle a developer can have. Sometimes it is — you get errors as you type, and indeed many code linters, which are a different type of analyzers do this. Some other times, this might be something we need to skip in favor of a bit slower feedback cycle; most commonly when someone is working and the work isn’t “done” yet. In other words — “I <em>know</em> I’m not ready, please don’t bother me, clippy”.</p> <p><img src="https://rng0.io/blog/why-we-need-developer-tools-for-security-and-not-security-tools-for-developers/10.jpg" alt="" /></p> <p>In this case, it’s more appropriate to approach the feedback cycle just as you’d approach development unit tests. <strong>In the same way unit tests run, you can run security tools</strong>.</p> <p>Only one big deal requirement here — run <em>fast</em>. This may become a big requirement for current generation tools based on heavy AST data flow analysis, and for some an impossible requirement as it may take these hours to complete a run. Even a few <em>minutes</em> may be too much.</p> <h2 id="precommit">Precommit</h2> <p>Other than unit tests, there is the staging step. In this step developers are reviewing their work and ready to stage the work into a codebase to be shared with others. In this staging step, we can use <a href="https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks">a git-hook</a>. <strong>A pre-commit hook will trigger evaluation of the material that’s about to be committed and possibly block a commit if there is a violation</strong>.</p> <h2 id="ci">CI</h2> <p>This about ends the story on the developer machine. <strong>The next step is to create a feedback cycle on the CI system itself</strong>. We’re in the developer realm but it also means such a given feedback cycle is now in the — several minutes to — tens of minutes time frame -.</p> <p>We’re also taking two new risks:</p> <p><img src="https://rng0.io/blog/why-we-need-developer-tools-for-security-and-not-security-tools-for-developers/11.jpg" alt="" /></p> <ol> <li>A security risk where a material we have may end up in foreign-owned central store (e.g. credit cards)</li> <li>A productivity risk where we’re competing with time shares from the CI system, and from fellow engineers to review our work manually, or to block other people’s work if the branching model is such that work in various stages can block others (dev branch, but no PR branches or no concept of “PR review”, for example)</li> </ol> <h2 id="what-about-threat-modeling">What about threat modeling?</h2> <p><strong>Whatever tool you’re running in your pipeline now, it should correlate with some concept of threat, and be able to assist in risk mitigation and its prioritization</strong>.</p> <p>Does this now mean that threat modeling becomes a bottleneck? If everything runs super fast, tools are being a great citizen in your CI and developer endpoints and deliver great security value.</p> <p>The answer is yes and no. Just like when software moved to BDD and TDD, and away from “QA on production”, and later ops moved to infrastructure as code (and still is moving) and away from “click to bring up a cluster”, a few roles became overloaded, and a few years-old processes needed a refresh or were killed altogether.</p> <p>Source: <a href="https://hangoutagile.com/popular-agile-methodologies-wave-1/">https://hangoutagile.com/popular-agile-methodologies-wave-1/</a></p> <p><img src="https://rng0.io/blog/why-we-need-developer-tools-for-security-and-not-security-tools-for-developers/12.jpg" alt="" /></p> <p>Aligning security processes with the TDD movement, as well as with the IaC movement, and cloud-native movement.</p> <p>There <em>is</em> a trend here — as software understanding improves, we’re able to create more software that pushes production and design of software to the natural place that it should live in — like any other form of engineering — design stage.</p> <p>Because we don’t experience “waste” in software engineering, other than the ever-illusive loss of time, we’ve never been really keen on pushing this so strongly. Compare that to a carpenter that keeps building furniture but never measures, delivers a cabinet just to see it’s crooked, and then goes back to rebuild, throwing away a lot of wood — this person <em>sees</em> the material being wasted.</p> <p>In fact, it’s inevitable that this role of the threat-modeler is going to move to be strained. Therefore it has to align with some kind of more agile process; as critics of STRIDE already saw — we need some common sense for where and how to apply threat modeling.</p> <p>And so, as you shape up your developer tools for security, you need to do the same for your <em>developer processes for security</em>.</p> <ul> <li>Run a threat modeling session in pre-planning, like you perform all other risk-assessment activities (capacity planning, architecture and others)</li> <li>Bring up and document <a href="http://swreflections.blogspot.com/2013/10/adding-appsec-to-agile-security-stories.html">evil user stories</a> (or “ <a href="https://ieeexplore.ieee.org/document/1306981?arnumber=1306981">misuse cases</a>” or “ <a href="https://rietta.com/blog/what-is-an-abuser-story-software/">abuser stories</a>”) to serve as the working cases</li> <li>For a more systematic approach for developing these stories (and of course based on the effort you can invest) — pick <a href="https://docs.microsoft.com/en-us/archive/msdn-magazine/2006/november/uncover-security-design-flaws-using-the-stride-approach#S2">STRIDE</a> or <a href="https://community.wardleymaps.com/t/tackling-security-with-wardley-maps/76">Wardly</a> models to guide you through a complex and risky system</li> <li>Tracking and applying these stories is a different kind of thing: some people go to extremes and use <a href="https://cucumber.io/docs/gherkin/">gherkin</a> to write, track, and (possibly) execute these, with an ode to BDD. However, it does require a significant investment with tooling and techniques to run efficiently and properly; and again it goes back to developer tools for security (ever ran Zap as a part of a pipeline and had to automate UI? how did you deal with crashes?)</li> </ul> <h2 id="summary">Summary</h2> <p>Some great <strong>open source</strong> developer-first and CI friendly security tooling that you should check:</p> <ul> <li><a href="https://github.com/SpectralOps/teller"><strong>Teller</strong></a> — Secret management for developers and production workflows <strong>(which we made!)</strong></li> <li><a href="https://github.com/tfsec/tfsec">tfsec</a> — Security scanner for your Terraform code <strong>(which we sponsor and love!)</strong></li> <li><a href="https://github.com/zegl/kube-score">kube-score</a> — Kubernetes object analysis with recommendations for improved reliability and security</li> <li><a href="https://github.com/quay/clair">Clair</a> — very familiar and in use by many other security tools — static analysis for containers</li> <li><a href="https://github.com/securego/gosec">gosec</a> — security checker for Golang</li> </ul> <p>And a few takeaways if you’re planning on integrating or building developer-friendly security tools:</p> <ol> <li><strong>For every security tool that you know, there is probably another emerging tool that takes a developer-first approach</strong>. If not that, you can at least make sure it has features that allow you to build a developer-first pipeline with the existing tools today. You should now know what to look for.</li> <li><strong>Identify your existing processes, know that once you shift-left some of them might start becoming bottlenecks</strong>. It’s a good time to experiment with lean and agile processes for your security planning too (e.g. threat modeling)</li> <li>If you intend to build tools for your pipeline, you know now what kind of trade-offs to take, and what to do and what not to do.<em>Originally published at</em> <a href="https://spectralops.io/blog/why-we-need-developer-tools-for-security-and-not-security-tools-for-developers/"><em>https://spectralops.io</em></a> <em>on April 19, 2021.</em></li> </ol> 8 Steps for Troubleshooting Your Rust Build Times Tue, 06 Oct 2020 00:00:00 +0000 Unknown https://rng0.io/blog/8-steps-for-troubleshooting-your-rust-build-times/ https://rng0.io/blog/8-steps-for-troubleshooting-your-rust-build-times/ <p>To be able to troubleshoot Rust compilation, we first need to understand in a rough cut how it works. The goal is to give you a map that will be easy to remember, and be operational rather than highly detailed and academic.</p> <p>Think of it as a kind of an 80/20 rule set for the first steps you should do when you are trying to handle a slow build that’s unacceptable. By the way, <strong>my focus here is on your development cycle and fast iteration</strong>, rather than a slow build in CI (but the tips here may be relevant for it as well).</p> <p>A number of interesting steps in the Rust build cycle are:</p> <ol> <li><strong>Take all crates and generate a</strong> <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph"><strong>DAG</strong></a> which is a convenient way to understand which crate depends on (or blocks the compilation of) another crate, and by that, for example — Rust will be able to understand which crates can run in parallel and use all machine cores</li> <li>If that’s not enough, <strong>divide each crate into</strong> <a href="https://doc.rust-lang.org/rustc/codegen-options/index.html#codegen-units"><strong>codegen units</strong></a> to let LLVM process them in parallel as well</li> <li>And lastly, <a href="https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation.html">incremental compilation</a> helps because Rust will save part of the compilation planning work from a previous build and reuse it in the next</li> </ol> <p><img src="https://rng0.io/blog/8-steps-for-troubleshooting-your-rust-build-times/1.png" alt="logo of llvm" /></p> <p>LLVM serves as a major infrastructure for Rust. While Rust performs compilation tasks on its side, some of those tasks <strong>are handed off to LLVM</strong> and as such, become a black box to us, users of Rust.</p> <p>Another important note is that LLVM is critical for speedy compilation, when <a href="https://lists.llvm.org/pipermail/llvm-dev/2020-May/141482.html">it goes slow, Rust goes slow</a> and that can happen as LLVM bakes in improvements and fixes. Rust being downstream to these changes, risks some of the changes as regression in compilation speed — sadly on the Rust side.</p> <p>LLVM is also critical to your compilation speed <strong>because Rust is delegating a good part of the job of optimizing code to it</strong>. To get a feel — run a debug build and then a release build. You should see around an order of magnitude (10x) difference between the performance of the two builds.</p> <p>We see static linking happening <strong>at the end of the build cycl</strong> e, which generates a nice and self-contained binary to use. While I find self-contained binaries extremely useful and am a big fan, it slows down each build (as opposed to dynamic linking when dependencies are loaded at runtime).</p> <p>And now, for the steps.</p> <h2 id="1-create-a-speedy-build-experience">1. Create a Speedy Build Experience</h2> <p>The first step to create a fast build is not to build at all. Using <a href="https://github.com/rust-analyzer/rust-analyzer">rust-analyzer</a> in your editor and development process gives you just this experience. This means that you’re implicitly running <code>cargo check</code> which is the first tip for anyone trying to optimize their build speeds. Most people want fast feedback, and you can get that feedback with <code>cargo check</code> instead of <code>cargo build</code>.</p> <h2 id="2-throw-hardware-at-it">2. Throw Hardware At It</h2> <p><img src="https://rng0.io/blog/8-steps-for-troubleshooting-your-rust-build-times/2.jpg" alt="picture of a broken bridge" /></p> <p>With that first tip out of the way, the second tip is: <em>before fixing a problem are you sure it really is a problem?</em>. <strong>Which means in our world of slow builds, are you sure your Rust build being slow is really a problem?</strong> For example, if you can throw more hardware at it at a reasonable price, this “problem” goes away; you’re robbed of the experience of actually fixing it but the result is the same, the problem goes away. In Rust, more cores <em>usually</em> lead to faster builds, so make sure you have a good amount of cores in your CPU.</p> <p>Rust <strong>has built in tools</strong> at your disposal that help you figure out what happens during the build process. We’re going to take a look at them, and order them by level of granularity and complexity (or investment) from your side.</p> <p>Our goal is to find a simple tool that lets you discover a big problem, solve it quickly and get big wins. But we’re not always that lucky, so we’ll also explore other tools and techniques that may be useful, but also may lead to <a href="https://en.wikipedia.org/wiki/Diminishing_returns">diminishing returns</a> like with many optimization tools.</p> <h2 id="3-big-things-build-slowly">3. Big Things Build Slowly</h2> <p><img src="https://rng0.io/blog/8-steps-for-troubleshooting-your-rust-build-times/3.jpg" alt="" /></p> <p>The first sign of a big build process is a big binary and big things take more time to build right?. And eyeballing a binary size is something we can do without tools.</p> <p>We already know that a big binary, with a reasonable or modest amount of <em>our</em> code means — <strong>lots of libraries that we <em>probably</em> use</strong>. We say <em>probably</em> because not always we understand the entire graph of dependencies that we use; most libraries take in other libraries as a dependency and so on.</p> <p>This leads us to want to figure out which crates we use and what is their effect on our build in terms of size (which is a good indicator of build bottlenecks per our thesis earlier).</p> <p><strong>A great tool for that is</strong> <code>cargo-bloat</code>, which is inspired from the same intuition I've described here <a href="https://github.com/google/bloaty">at Google</a> (by the way: I've built such a tool for Go called <a href="https://github.com/jondot/goweight">goweight</a>). <code>cargo-bloat</code> works with your <code>cargo</code> and any existing build:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo install cargo-bloat</span> </span></code></pre> <p>Starting with a detailed view, and listing the top 10 results, gives us:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo bloat<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>release</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>n</span> 10</span> </span></code></pre> <p><img src="https://rng0.io/blog/8-steps-for-troubleshooting-your-rust-build-times/4.png" alt="" /></p> <p>While this is great for detailed analysis, what we want is the “big targets”, so let’s analyze by crates:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo bloat<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>release</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>crates</span></span> </span></code></pre> <p><img src="https://rng0.io/blog/8-steps-for-troubleshooting-your-rust-build-times/5.png" alt="" /></p> <p>There we have it. According to our theory, <code>regex</code>, and <code>libgit2</code> could be slowing our build because they result in taking a big chunk of our binary (and as a spoiler: this theory verifies later). At this stage, if you discover a big library that you use, and don't need, or can replace with something simpler -- you can do that and re-run the build. If you find that it shortens your build times and you're happy -- <strong>you incidentally found a build optimization unicorn</strong> -- with minimal effort invested and maximal gain. You've fixed the problem!</p> <h3 id="4-take-a-deeper-look">4. Take a Deeper Look</h3> <p>In the case where we’re not so lucky, and unicorns don’t exist (sorry, unicorn fans!), we’re left with doing <strong>real build process analysis</strong>. We have a couple of ways to do that.</p> <p>To get a view of how long everything takes in your build process, we use <code>-Z time-passes</code>:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> RUSTFLAGS=<span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>-Z time-passes<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span> cargo build</span> </span></code></pre> <p><img src="https://rng0.io/blog/8-steps-for-troubleshooting-your-rust-build-times/6.png" alt="" /></p> <p>You’ll see a lot of data streaming fast. You get a detailed breakdown of the build process, per crate, and <strong>per compilation stage</strong>. By glancing at this you’ll be able to get an impression of the two most time consuming tasks: codegen and tasks that deal with LLVM handoff and optimization.</p> <h3 id="a-magical-view-for-your-build">A magical view for your build</h3> <p>Actually, the best trade-off between high detail and high productivity, is a built in <code>cargo</code> flag, <a href="https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#timings">called</a> <code>timings</code>. It's such a great tool, it's almost <em>magical</em> and in fact, I use it exclusively, even for smaller troubleshooting session.</p> <p>If you run this:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo build<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>Z</span> timings</span> </span></code></pre> <p>You’ll get this:</p> <p><img src="https://rng0.io/blog/8-steps-for-troubleshooting-your-rust-build-times/7.png" alt="" /></p> <p>It is an interactive graph showing the build breakdown graph, where you can see times, parallelism, and deeply and effectively investigate why a build is slow. To me, being familiar with profilers in many other languages, it was <em>mindblowing</em>.</p> <p>And then you get this which <strong>graphically shows you the bottlenecks</strong>, spikes and CPU usage as the build progresses:</p> <p><img src="https://rng0.io/blog/8-steps-for-troubleshooting-your-rust-build-times/8.png" alt="" /></p> <p>And here’s a larger, more complex build process:</p> <p><img src="https://rng0.io/blog/8-steps-for-troubleshooting-your-rust-build-times/9.png" alt="" /></p> <p>You also get a nice breakdown that looks the same on a first glance, but on a second glance contains important information: <strong>the <em>default features selected</em> for every crate.</strong> This is actually your first experimentation opportunity. If you are not familiar with crate features — every crate you use can come with features that you can enable or disable, and then — it comes with the default set of features.</p> <p>This means you can disable features you don’t need or use to get big compilation speed wins; in fact you should do this any way — <strong>keep a nice culture in your team of hand picking the features you use.</strong></p> <p><img src="https://rng0.io/blog/8-steps-for-troubleshooting-your-rust-build-times/10.png" alt="" /></p> <p>If it matters, keep in mind that <code>cargo build -Z timings</code> is <em>unstable</em>, to read more about it <a href="https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#timings">look here</a>.</p> <h2 id="5-replace-your-linker">5. Replace Your Linker</h2> <p>While I don’t want to be too nostalgic (I’m tempted to), linking used to be a process you do by hand, or at least a process you’re very aware of, and used to be a common ceremony if you were doing assembler or C more than 20 years ago. Today it’s mostly transparent, or at least the experience of linking is.</p> <p>A linker produces <strong>the final compilation output</strong> from various compiler object outputs. The kinds and types of compiler object output varies between compilers, languages and more.</p> <p>We mostly have the linker produce a binary in a form of a dynamic library, or an executable.</p> <p>In Rust, on <em>some</em> platforms (such as OSX) linking <strong>take a considerable amount</strong> of time from the general build process — too long. So this step is a good place to look for optimizations.</p> <p>In addition, we have link-time optimization (LTO) which <em>release</em> build configuration uses to perform optimizations across code units, and is typically as expensive as it sounds.</p> <h3 id="what-s-in-a-faster-linker">What’s in a Faster Linker?</h3> <p>Why is replacing a linker faster? to get an impression, here’s the notes from <code>zld</code> <a href="https://github.com/michaeleisel/zld">'s README</a> :</p> <ul> <li>Using <strong>Swiss Tables</strong> instead of STL for hash maps and sets</li> <li><strong>Parallelizing</strong> in various places (the parsing of libraries, writing the output file, sorting, etc.)</li> <li>Optimizations around the <strong>hashing of strings</strong> (caching the hashes, using a better hash function, etc.)</li> </ul> <h3 id="linux">Linux</h3> <p>In Linux the work is pretty much cut out for you by installing and using <code>lld</code> which is a much faster linker for Rust to use:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> sudo apt-get install lld</span> </span></code></pre> <p>Then, place a cargo config file in <code>.cargo/config</code>:</p> <pre data-lang="toml" class="language-toml z-code"><code class="language-toml" data-lang="toml"><span class="z-source z-toml"><span class="z-punctuation z-definition z-table z-begin z-toml">[</span><span class="z-meta z-tag z-table z-toml"><span class="z-entity z-name z-table z-toml">target</span><span class="z-punctuation z-separator z-table z-toml">.</span><span class="z-entity z-name z-table z-toml">x86_64-unknown-linux-gnu</span></span><span class="z-punctuation z-definition z-table z-end z-toml">]</span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">linker</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>/usr/bin/clang<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">rustflags</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-punctuation z-definition z-array z-begin z-toml">[</span><span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>-Clink-arg=-fuse-ld=lld<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span><span class="z-punctuation z-separator z-array z-toml">,</span> <span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>-Zshare-generics=y<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span><span class="z-punctuation z-definition z-array z-end z-toml">]</span> </span></code></pre> <p>Run your builds as usual and notice the improvement:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo build</span> </span></code></pre> <h3 id="macos">macOS</h3> <p>This solution didn’t exist for a long while (you couldn’t do anything about linking speeds) for macOS, <strong>but recently we’ve got</strong> <code>zld</code> <strong>support</strong> and we can use it instead:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> brew install michaeleisel/zld/zld</span> </span></code></pre> <p>And now you can drop this in your cargo config file in <code>.cargo/config</code>:</p> <pre data-lang="toml" class="language-toml z-code"><code class="language-toml" data-lang="toml"><span class="z-source z-toml"><span class="z-punctuation z-definition z-table z-begin z-toml">[</span><span class="z-meta z-tag z-table z-toml"><span class="z-entity z-name z-table z-toml">target</span><span class="z-punctuation z-separator z-table z-toml">.</span><span class="z-entity z-name z-table z-toml">x86_64-apple-darwin</span></span><span class="z-punctuation z-definition z-table z-end z-toml">]</span> </span><span class="z-source z-toml"><span class="z-meta z-tag z-key z-toml"><span class="z-entity z-name z-tag z-toml">rustflags</span></span> <span class="z-punctuation z-definition z-key-value z-toml">=</span> <span class="z-punctuation z-definition z-array z-begin z-toml">[</span><span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>-C<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span><span class="z-punctuation z-separator z-array z-toml">,</span> <span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>link-arg=-fuse-ld=/usr/local/bin/zld<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span><span class="z-punctuation z-separator z-array z-toml">,</span> <span class="z-string z-quoted z-double z-basic z-toml"><span class="z-punctuation z-definition z-string z-begin z-toml">&quot;</span>-Zshare-generics=y<span class="z-punctuation z-definition z-string z-end z-toml">&quot;</span></span><span class="z-punctuation z-definition z-array z-end z-toml">]</span> </span></code></pre> <p>This alone has cut my build times in <em>half</em>. <strong>HALF</strong>!</p> <p>Run your builds as usual and notice the improvement:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cargo build</span> </span></code></pre> <h3 id="global-or-local-config">Global Or Local Config?</h3> <p>I recommend setting your <code>~/.cargo/config</code> globally to enjoy faster build time around your workstation, however if you're uneasy about replacing a linker for what ever reason you can always have it local in every project's <code>.cargo</code> folder.</p> <p>Either way, <strong>make sure production builds are made and cut only from your CI system</strong>, which doesn’t have any of these tweaks (just to be safe).</p> <h2 id="6-crate-shattering">6. Crate Shattering</h2> <p><img src="https://rng0.io/blog/8-steps-for-troubleshooting-your-rust-build-times/11.jpg" alt="crate shattered" /></p> <p>When you’re out of low-hanging fruit, you’re left with old fashioned grunt work. This means taking a long look at how your project is architecture and try to guess how to break or merge it in different ways.</p> <p>One of the techniques in the Rust world <strong>is fondly named <em>crate shattering</em></strong> which is the art of taking a project and breaking it up into crates using the workspace feature of Cargo, or taking an existing crate and breaking it apart to smaller ones. All this, in an attempt to play into Rust’s way of parallelizing builts.</p> <p>Why <em>art</em>? because <strong>sometimes crate shattering causes the invest effect</strong> — challenging inlining and creating duplicate work for the Rust compiler (such as generic translation which is done per-crate); all this without a chance of a reasonable human effort of understanding the build intricacies and compensating for those.</p> <p>We can’t stand a chance of imagining all of the compiler complexities and optimizations and figure out “oh, that file <em>has</em> to live in this crate then”.</p> <p>There’s a <strong>pretty old</strong> <a href="https://www.reddit.com/r/rust/comments/3tco32/shattering_a_crate_in_pursuit_of_compile_times/"><strong>reddit thread</strong></a> <strong>putting some data behind crate shattering</strong>, but I say this — try to design your project with multiple crates in mind for the sake of software engineering, and hope for the best. This way: there are already wins that you’ll gain in productivity by designing a great architecture for your project, and there’s a <em>chance</em> of wins in compile times.</p> <h2 id="7-conditionally-compile-a-go-fast-feature">7. Conditionally Compile a “Go Fast” Feature</h2> <p>Here’s a trick I’ve developed on my own, and I have indications in the Rust documentation for it but haven’t really seen it in the wild.</p> <p>It goes like this; if you’re confident that there is some codebase <em>in development time</em> that you don’t really need, <strong>for example some kind of complex encryption algorithms — make that codebase a</strong> <a href="https://doc.rust-lang.org/cargo/reference/features.html"><em><strong>feature</strong></em></a> in the sense that it’s a special “go fast” feature — or any other name that you like, that when you activate this feature will conditionally compile it, and will leave it out of the build and therefore out of your compile times.</p> <p>I’ve spotted it in the Rust docs, but still I’m not sure they meant what I mean, it’s called <code>go-faster</code> in this shot:</p> <p><img src="https://rng0.io/blog/8-steps-for-troubleshooting-your-rust-build-times/12.png" alt="" /></p> <h2 id="8-use-small-new-projects-on-the-side">8. Use Small New Projects on the Side</h2> <p>When all fails, what works best is to just iterate quickly on an idea, a POC or a crate in a fresh folder, with this great feeling of starting from scratch: naturally a small amount of dependencies, small amount of source files, and low complexity for the compiler as a result.</p> <p>When you’re done iterating fast, you can migrate this code into the monster of a project you’ve been suffering slow build times in.</p> <p>This works so well, and is so refreshing, both in code-completion, code checks and compilation speeds that I build new features this way exclusively when I can.If you like developer tools, productivity and security — we’re looking for a great chat, and/or great people to join us at <strong>Spectral</strong>! drop us a line: <a href="mailto:hello@spectralops.io">hello@spectralops.io</a><em>Image credits:</em></p> <ul> <li><em>“Broken bridge” by Eva the Weaver is licensed with CC BY-NC-SA 2.0. To view a copy of this license, visit</em> <a href="https://creativecommons.org/licenses/by-nc-sa/2.0/"><em>https://creativecommons.org/licenses/by-nc-sa/2.0/</em></a></li> <li><em>“The big truck at Sparwood 2010” by Gord McKenna is licensed with CC BY-NC-ND 2.0. To view a copy of this license, visit</em> <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/"><em>https://creativecommons.org/licenses/by-nc-nd/2.0/</em></a></li> <li><em>“Discarded Boxes” by cbmd is licensed with CC BY-NC-ND 2.0. To view a copy of this license, visit</em> <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/"><em>https://creativecommons.org/licenses/by-nc-nd/2.0/</em></a></li> </ul> Building Rust on Multiple Platforms Using Github Fri, 21 Feb 2020 00:00:00 +0000 Unknown https://rng0.io/blog/building-rust-on-multiple-platforms-using-github/ https://rng0.io/blog/building-rust-on-multiple-platforms-using-github/ <p><img src="https://rng0.io/blog/building-rust-on-multiple-platforms-using-github/1.png" alt="" /></p> <p><a href="https://github.com/features/actions">Github Actions</a> make a compelling option if one of your main challenges are to build binaries over wide array of platforms such as Linux, Windows, and macOS.</p> <p>While the difference between a Linux and a given macOS in terms of build aren’t significant (we’ll see how they <em>can</em> become significant actually), Windows and especially how to get to a nice Rust build on Windows using MSVC require an investment on your part.</p> <p>For the most part the Rust <a href="https://github.com/rust-embedded/cross">cross-compilation toolchain</a> holds its weight well when you’re looking to build for various architectures, but we’ll see a few cases where you’re on your own with Github Actions.</p> <p><img src="https://rng0.io/blog/building-rust-on-multiple-platforms-using-github/2.jpg" alt="" /></p> <h2 id="rust-and-windows">Rust and Windows</h2> <p>The seamless option when building Rust programs for Windows is the <a href="https://gnutoolchains.com/">GNU toolchain for Windows</a>, you might be familiar with <a href="https://www.cygwin.com/">Cygwin</a> as well.</p> <p>The Rust binary that results from a GNU toolchain build is bulky and slow but there should be very little compatibility issues or surprises. If being bulky and slow is a deal breaker for you (I believe it usually is), then you would have to build for Windows “for real”, with the MSVC toolchain — which is what many native Windows programs build against.</p> <p>While the official docs are slim on preparing a <a href="https://doc.rust-lang.org/edition-guide/rust-2018/platform-and-target-support/msvc-toolchain-support.html">MSVC C/C++ toolchain</a>, Rust really does work seamlessly with the MSVC toolchain. If you’re out of using Windows for a long while, it’ll be awkward to prepare a fully working Windows developer machine from scratch.</p> <p>To avoid going through that, I recommend using Github Actions, which has great Windows support and you won’t even have to remember how to prepare all of the Windows tooling that you need.</p> <p>You can read more about this in <a href="https://users.rust-lang.org/t/why-do-i-need-microsoft-c-build-tools">“Why I need Microsoft C++ Build Tools”</a> and <a href="https://users.rust-lang.org/t/what-build-tool-for-windows-is-required/20678/7">“What Build Tools for Windows are Required”</a>.</p> <p>To build on Windows locally (some times it makes sense to get faster dev cycles or while prototyping), I personally prefer not to touch a Windows UI or installing Windows at all, and I use <a href="https://www.vagrantup.com/downloads.html">Vagrant</a> on my macOS, or more specifically <a href="https://github.com/Baune8D/vagrant-vs2017-devbox">vagrant-vs2017-devbox</a>.</p> <p>There’s also the slightly more cooked <a href="https://app.vagrantup.com/mcandre/boxes/vagrant-windows-rust">vagrant-windows-rust</a>.</p> <p><img src="https://rng0.io/blog/building-rust-on-multiple-platforms-using-github/3.png" alt="" /></p> <p>With Vagrant, you just <code>vagrant up</code> and do everything you need over SSH, including building, transferring assets and fetching a built binary from the Windows guest. It really works.</p> <p>All in all, if you can afford it, always opt to use a CI service that has Windows support that can take all this pain away from you.</p> <h2 id="rust-and-linux-with-musl">Rust and Linux With musl</h2> <p><img src="https://rng0.io/blog/building-rust-on-multiple-platforms-using-github/4.png" alt="" /></p> <p>If you want to build a binary on Linux and have it use standard C libraries (which it eventually needs), you have two popular options: GNU and musl.</p> <p>You can dive deeper by reading this <a href="https://www.reddit.com/r/linuxmasterrace/comments/41q2m9/eli5_what_is_musl_and_glibc/">ELI5 (explain like I’m five)</a> about the difference between the GNU and musl C lib variants. But to make a long story short, if you would be considering <em>musl</em> over the much more popular and “default” GNU, it would be:</p> <ul> <li>Compatibility — not all Linux distros has GNU</li> <li>Performance — <a href="http://www.etalabs.net/compare_libcs.html">while this is still arguable</a> about which is <em>faster</em>, musl is definitely more lightweight</li> <li>Static linking — this is probably the most important aspect of producing a fully <a href="https://doc.rust-lang.org/edition-guide/rust-2018/platform-and-target-support/musl-support-for-fully-static-binaries.html">portable and static Rust binary</a>. Right now <em>you can only do that with musl</em></li> </ul> <p>Building with musl, as of this writing, from my experience — is going off the beaten path. To go the extra mile for having a statically compiled and independent binary you’ll have to find or build your own development workflow that works for you.</p> <p>While it’s no longer such a big deal (you can use <a href="https://hub.docker.com/r/ekidd/rust-musl-builder/">rust-musl-builder</a>), you will have to do the ceremonies in <a href="https://github.com/emk/rust-musl-builder/blob/master/Dockerfile">this Dockerfile</a> with every new environment you would need to customize.</p> <h2 id="github-actions-and-musl">Github Actions and musl</h2> <p>One such environment would be Github Actions. To have a musl based build environment that can support <em>nightly</em>, <em>OpenSSL</em> and a few more real-world use cases, you’d have to skip the standard <a href="https://github.com/actions-rs/toolchain">Rust Toolchain Github Action</a> and build one yourself, depending on your requirements.</p> <p>While you can still use these standard actions like so:</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">name</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">Install Rust</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">uses</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">actions-rs/toolchain@v1</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">with</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">toolchain</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">nightly</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">target</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">x86_64-unknown-linux-musl</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">default</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-constant z-language z-boolean z-yaml">true</span> </span></code></pre> <p>You risk chasing after errors like:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">Internal error occurred: Failed to find tool. Is `musl-gcc` installed? </span></code></pre> <p>Which might be solvable easily, or even indicate a wrong way to use these actions, but all-in-all there is currently little documentation or ways to iterate quickly on top of the Github Actions environment sandbox.</p> <p><img src="https://rng0.io/blog/building-rust-on-multiple-platforms-using-github/5.png" alt="" /></p> <p>Huh?</p> <p>There is an invisible line that you cross when you decide to go with more power and brewing your own to hedge the risk of losing time by adapting existing tools, and in this case I recommend crossing it.</p> <p>I’ve used <a href="https://github.com/juankaram/rust-musl-action">rust-musl-action</a> as a basis for my custom action, and working around how Github modeled Actions, it took a while to get all of the edge cases and use cases of my large (and funky, when it comes to using FFI) codebase to smooth out.</p> <p>To use your own custom action you can <a href="https://github.com/juankaram/rust-musl-action">copy the rust-musl-action</a> one into your <code>.github/actions</code> folder and do this in your <code>workflow</code>:</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">name</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">Rust w/musl</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">uses</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">./.github/actions/rust-musl-builder</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">env</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">BUILD_TARGET</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">x86_64-unknown-linux-musl</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">with</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">args</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">./build-default.sh</span> </span></code></pre> <p>From here on, you can package anything you want in the <code>rust-musl-builder</code> docker, and iterate around it until you can get a healthy binary building.</p> <h2 id="github-actions-matrix-strategy">Github Actions Matrix Strategy</h2> <p>This is <a href="https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstrategy">the killer feature</a> that Github Actions has. When it works, and when you’re not into customizing things heavily like we’re doing here — it’s magic.</p> <p>If you’re building for:</p> <ul> <li>OS: Windows, Linux (GNU), macOS</li> <li>Node: various ABIs (12.x, 13.x, electron) via <a href="https://github.com/neon-bindings/neon">neon</a></li> </ul> <p>As long as you’re staying within reasonable configuration — it will be a minimal amount of fussing around and you’re going to get the matrix equivalent of <code>OS x Node</code> binaries baked and ready to serve.</p> <p>But if you want <code>musl</code> you'll have to have that one annoying exception. So here's how I model my own workflow to support using <code>musl</code>:</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">build_musl</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">steps</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml">.. your musl-specific job ..</span> </span><span class="z-source z-yaml"> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">build_rest</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">strategy</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">matrix</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">os</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-meta z-flow-sequence z-yaml"><span class="z-punctuation z-definition z-sequence z-begin z-yaml">[</span><span class="z-string z-unquoted z-plain z-in z-yaml">windows-latest</span><span class="z-punctuation z-separator z-sequence z-yaml">,</span> <span class="z-string z-unquoted z-plain z-in z-yaml">macos-latest</span><span class="z-punctuation z-definition z-sequence z-end z-yaml">]</span></span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">runs-on</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">${{ matrix.os }}</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">steps</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml">.. use the regular Rust Github Action here ..</span> </span></code></pre> <p>This allows your to capitalize on everything that the magic that Github Actions give you by staying on the regular ways of doing things for macOS and Windows, but have a specialization for Linux/musl.</p> <h2 id="keep-your-ci-code-cross-platform">Keep Your CI Code Cross Platform</h2> <p>You might be tempted to think Windows and Unix are compatible these days, and if you lower your head by not handling, for example, the issues of <a href="https://shapeshed.com/writing-cross-platform-node/#paths">slashes</a> and <a href="https://en.wikipedia.org/wiki/Newline">newlines</a> it’ll go away. Well, if you’re doing anything a little bit involved it’ll blow up.</p> <p>Here’s a few indicators for where to look:</p> <ul> <li>Composing paths to find and package your binaries</li> <li>Scripts that extract the current version with a regex and a newline-handling regex (hint: it might not match)</li> <li>You’re not in a <em>real</em> shell in Windows. When in doubt be minimal and use either a well behaving (in terms of cross platform) language such as Node.js or Python to do what you’d otherwise do in a shell. Make one of those your main CI driver</li> </ul> <h2 id="summary">Summary</h2> <p>For your reference, here’s all of the extra reading in one place:</p> <ul> <li>Rust’s <code>cross</code>, the <a href="https://github.com/rust-embedded/cross">cross-compilation toolchain</a></li> <li><a href="https://gnutoolchains.com/">GNU Toolchain for Windows</a> and <a href="https://www.cygwin.com/">Cygwin</a></li> <li>Rust’s MSVC <a href="https://doc.rust-lang.org/edition-guide/rust-2018/platform-and-target-support/msvc-toolchain-support.html">support</a></li> <li><a href="https://users.rust-lang.org/t/why-do-i-need-microsoft-c-build-tools">“Why I need Microsoft C++ Build Tools”</a></li> <li><a href="https://users.rust-lang.org/t/what-build-tool-for-windows-is-required/20678/7">“What Build Tools for Windows are Required”</a></li> <li>A <a href="https://github.com/Baune8D/vagrant-vs2017-devbox">Vagrant Windows / Rust</a> all-in-one dev box</li> <li><a href="https://www.reddit.com/r/linuxmasterrace/comments/41q2m9/eli5_what_is_musl_and_glibc/">ELI5 (explain like I’m five) for what’s musl and glibc</a></li> <li>Building <a href="https://doc.rust-lang.org/edition-guide/rust-2018/platform-and-target-support/musl-support-for-fully-static-binaries.html">portable and static Rust binary</a></li> <li>Docker for building with musl on Rust — <a href="https://hub.docker.com/r/ekidd/rust-musl-builder/">rust-musl-builder</a></li> <li>The <a href="https://github.com/actions-rs/toolchain">Rust Toolchain Github Action</a> and <a href="https://github.com/juankaram/rust-musl-action">rust-musl-action</a> as a custom musl one</li> <li>Writing <a href="https://shapeshed.com/writing-cross-platform-node/#paths">cross platform Node.js</a></li> </ul> 4 New Theme Based React UI Toolkits and Why It’s Going To Change How You Think Wed, 29 Jan 2020 00:00:00 +0000 Unknown https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/ https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/ <p><code>styled-components</code> created a new standard for how we do CSS-in-JS. It introduced a sweet spot between writing CSS and React components, and it looked like this:</p> <p><img src="https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/1.png" alt="" /></p> <p>With this new API you could:</p> <ul> <li>Use something that looks like CSS, using the new <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals">template literal</a> feature from recent ECMAScript versions</li> <li>Create atomic-feeling React components</li> <li>Still preserve flexibility with access to <code>props</code> and use them through out the template literal freely</li> </ul> <p><img src="https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/2.png" alt="" /></p> <p>Made for a very low-friction developer experience, that was widely adopted, and then widely imitated (in a good way). We now have the same <code>styled</code> API in <a href="https://emotion.sh/docs/introduction">Emotion</a> and others.</p> <p>From here on, styled-components evolved to include <a href="https://styled-components.com/docs/advanced#theming">themes</a> and a <code>ThemeProvider</code> as well as a rich <a href="https://styled-components.com/ecosystem">ecosystem</a>.</p> <p>styled-components is considered to be one of the most popular libraries for doing CSS-in-JS, and with <a href="https://styled-components.com/releases#v5.0.0">version 5.0.0</a> just out in 2020, it catches up on performance with no API breaking changes.</p> <h2 id="emotion">Emotion</h2> <p><a href="https://emotion.sh/docs/introduction">Emotion</a> is another great library for your CSS-in-JS. If we avoid comparing it to styled-components (there are many comparisons already), it seemed to have settled on a niche of infrastructure for component libraries. Though I’m not sure as to why it did, I’m guessing that for a long while, it was considered to have first-class performance, while styled-components wasn’t (this is not the case anymore with styled-components 5.0.0).</p> <p>For this reason, many, if not all of the most important UI frameworks were built on Emotion, and not on styled-components.</p> <p>Largely, Emotion and styled-components have similar but not identical API and developer experience. If you want, both have the <code>styled</code> API, and using <code>styled</code> is a safe way to build something that can work with both:</p> <p><img src="https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/3.png" alt="" /></p> <p>But, and this is where our story starts to become more interesting, Emotion had put emphasis (and bet) on their now widely-known <a href="https://emotion.sh/docs/css-prop">css prop</a>.</p> <p>They state that their primary (better?) way to style components is through a special <code>css</code> prop that does a few things behind the scenes:</p> <p><img src="https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/4.png" alt="" /></p> <p>A few notes about the <code>css</code> prop (from Emotion docs):</p> <ul> <li>Similar to the style prop, but also has support for auto vendor-prefixing, nested selectors, and media queries.</li> <li>Allows developers to skip the styled API abstraction and style components and elements directly.</li> <li>The css prop also accepts a function that is called with your theme as an argument allowing developers easy access to common and customizable values.</li> <li>Reduces boilerplate when composing components and styled with emotion.</li> <li>Server side rendering with zero configuration.</li> <li>Theming works out of the box.</li> </ul> <p>Obviously, although the <code>css</code> prop appear to be inline with your component, its contents are compiled, extracted and put as a standard CSS block.</p> <p>The magic, in this example is done by the <code>jsx</code> pragma at the top. With it, React creates new components with the Emotion <code>jsx</code> function, which is a custom way to create React components, with the relevant css prop logic applied.</p> <h2 id="performance-is-no-longer-an-issue">Performance Is No Longer an Issue</h2> <p>Performance was a key differentiator between CSS-in-JS libraries for a long while. Well it isn’t an issue anymore:</p> <p><img src="https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/5.png" alt="" /></p> <p>You might want to look at the <a href="https://medium.com/styled-components/announcing-styled-components-v5-beast-mode-389747abd987">styled-components 5.0.0</a> for a nice comparison.</p> <p>With that out of the equation, we can have an honest look at the abstractions around UI frameworks or toolkits that were created using both Emotion and styled-components with a material view of what developer experience they supply us with.</p> <h2 id="flexbox-and-evergreen">Flexbox and Evergreen</h2> <p>We’re lucky to be living in a <a href="https://caniuse.com/#feat=flexbox">Flexbox and Evergreen</a> world today. This means, that for the most part we can use Flexbox and get away with handling browser layout quirks. Which also means we have a huge burden of hacking around browser support and bridging incompatible rendering on our own, lifted from our backs.</p> <p>In turn this creates less code, less hacks, and more opportunity for CSS-in-JS frameworks to be a legitimate replacement for CSS — we no longer need that demanding skillset of “knowing CSS”, in the sense that knowing CSS was really about knowing the differences between browsers and how to bridge those creatively (read: find hacks) and in a reliable manner.</p> <p>This also puts a lot of the classic <a href="https://960.gs/">grid systems</a> largely out of a job. Building grid system with Flexbox is a trivial matter today.</p> <h2 id="the-beginning-of-an-idea">The Beginning of an Idea</h2> <p>Before we move on, I’d like to plant an idea in your mind. We said that the <code>styled</code> interface also allows using <code>prop</code> like any other React component. So basically this:</p> <p><img src="https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/6.png" alt="" /></p> <p>And now, if we wanted to use a theme variable, having that a theme is supplied as a prop named <code>theme</code> how about this?</p> <p><img src="https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/7.png" alt="" /></p> <p>The important bit of information is that we call our background prop <code>bg</code>, so how about we create this function?</p> <p><img src="https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/8.png" alt="" /></p> <p>This function builds <em>theme accessor</em> functions, and now, we can use it to have a bit more of an elegant approach in our <code>styled</code> clause:</p> <p><img src="https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/9.png" alt="" /></p> <p>And this is the idea I wanted you to have in mind. From here, you can create a wide range of tools: theme getters, variants, responsive styles and more, all based on the same principle.</p> <p>In fact, many of the next-generation libraries we’re going to look at next use this idea as their basic building block.</p> <h2 id="theme-oriented-styling">Theme Oriented Styling</h2> <p>What if we take our idea further? Say we have this block:</p> <p><img src="https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/10.png" alt="" /></p> <p>There’s no such color <code>bg</code>, but if we already go through the process of "compiling" template literals from what <em>looks like</em> CSS to actual CSS, could we tweak our infrastructure to replace <code>bg</code> from a name to an actual value in our theme?</p> <p>For this to happen we need to:</p> <ol> <li>Have a theme ready in our context and available for every component (we need to know where to get the theme from)</li> <li>Where does the actual value for <code>bg</code> sit? is this the background for <em>button</em>? is this just one of a set of agreed-upon colors?</li> <li>Have an idea of what actually is a theme</li> </ol> <p>So basically if we answer for (3) we get free answers for our first two questions.</p> <p>First attempt:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">theme</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">bg</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>#fff<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"></span><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></code></pre> <p>Second attempt, a bit more verbose:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">theme</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">background</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">bg</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>#fff<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"></span><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></code></pre> <p>Now I need to “teach” my infrastructure code that all CSS props that deal with <code>background</code> need to key into the <code>.background</code> prop in our theme, always. Nice idea, right?</p> <p>But, if you notice we’re dealing with buttons, we could also do this:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">theme</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">buttons</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">one</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">background</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>#fff<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span> </span></span></span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"></span><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></code></pre> <p>And if we could, with a bit of imagination, use something like this instead of our <code>styled</code> interface:</p> <p><img src="https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/11.png" alt="" /></p> <p>And we <em>hope</em> someone is listening in to this <em>variant</em> prop, and pulls the correct style from the <em>buttons</em> section, and injects it into our styling for our particular button.</p> <p>Well, surprise! all these ideas, and more, are part of what I call next-gen CSS-in-JS frameworks.</p> <p>But all these ideas need <em>structure</em>, which if we think about where everything starts from — this structure exist in our modeling of a <em>theme</em>.</p> <p>This is where the <a href="https://system-ui.com">system-ui</a> theme <a href="https://system-ui.com/theme">specification</a> comes into play. It is <em>somewhat</em> widely adopted, and as of this writing there’s not much deviation from this “standard”, which is good news.</p> <p>Here’s an example:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> example fontSizes scale as an array</span> </span><span class="z-source z-ts"><span class="z-entity z-name z-label z-ts">fontSizes</span><span class="z-punctuation z-separator z-label z-ts">:</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span> </span></span><span class="z-source z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-constant z-numeric z-decimal z-ts">12</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">14</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">16</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">20</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">24</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">32</span> </span></span><span class="z-source z-ts"><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">]</span></span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> example colors object</span> </span><span class="z-source z-ts"><span class="z-entity z-name z-label z-ts">colors</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">blue</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>#07c<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">green</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>#0fa<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> example nested colors object</span> </span><span class="z-source z-ts"><span class="z-entity z-name z-label z-ts">colors</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">blue</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>#07c<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">blues</span><span class="z-punctuation z-separator z-label z-ts">:</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>#004170<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>#006fbe<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>#2d8fd5<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>#5aa7de<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">]</span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p>The theme specification takes care of:</p> <ul> <li>Order and convention. With this we know where colors go, where spaces go and so on</li> <li>Context. With a well-known spec, we can build tools (and people did) to apply proper understanding of what React component we’re trying to theme, and to pull the right style from a theme</li> <li>CSS-prop to theme-prop mapping. The spec determines that, for example, the <code>sizes</code> theme prop apply to <code>width, height, min-width, max-width, min-height, max-height</code> CSS props</li> </ul> <p>And this is how we use it:</p> <p><img src="https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/12.png" alt="" /></p> <p>More on the <code>sx</code> prop later, but the gist of it is that <code>width</code> is now super-powered with our theme infrastructure, <em>because</em> we know a CSS <code>width</code> is mapped to our theme <code>sizes</code> prop, we know it indexes into the <code>0</code> and <code>2</code> positions which may resolve to, say, ‘768px’ and ‘1070px’.</p> <p>Why the array? Because we are responsive by default. In this example, <code>[0,2]</code> relate to mobile/desktop respectively (you can apply more breakpoints).</p> <h2 id="the-road-to-design-systems">The Road to Design Systems</h2> <p>If we have a theme specification, that locks in our style guide, <a href="https://medium.com/eightshapes-llc/tokens-in-design-systems-25dd82d58421">design tokens</a> and lets us centrally apply a policy around our styles, then we have much of the work done for a pretty good design system.</p> <p>The kind of design system that’s managed centrally, in one place that is shared between engineering and design needs to have very transparent and comprehensible concepts — it needs to be almost as simple as a JSON file, and this is to a large degree what the theme specification represents.</p> <p>You can see part of what you can do <a href="https://theme-ui.com/demo">here</a>.</p> <h2 id="converging-concepts">Converging Concepts</h2> <p>When we look at next-gen UI frameworks like <a href="https://theme-ui.com/">Theme UI</a>, <a href="https://rebassjs.org/">Rebass</a>, <a href="https://chakra-ui.com/">Chakra</a>, and others, we see a few concepts that are very similar, and luckily enough are converging so we don’t have to learn many different things.</p> <h2 id="style-props">Style props</h2> <p>Style props give us <a href="https://en.wikipedia.org/wiki/HTML#HTML_3">HTML 3.0</a> back again (kidding!):</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">div</span> <span class="z-variable z-other z-readwrite z-ts">mr</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>30px<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span> <span class="z-variable z-other z-readwrite z-ts">width</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>200px<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></code></pre> <p>Here <code>mr</code> is a shorthand for <code>margin-right</code> and <code>width</code> is CSS <code>width</code>, and not your regular HTML element (HTML 3.0ish) width.</p> <p>The great advantage that style props bring is hackability and familiarity. You can use a component and <em>not</em> factor all of the edge cases into your component. This helps you create a component library that’s more resilient to change, and for example, you don’t have to make components incorporate logic to care of their spacing (margin, padding) in order to lay themselves out in a page.</p> <h2 id="variants">Variants</h2> <p>It seems that most frameworks now add the concept of a <em>variant</em>, which looks like this:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">div</span> <span class="z-variable z-other z-readwrite z-ts">variant</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>button<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></code></pre> <p>If I could pick a concept that already exists in CSS that you are familiar with, it would be to say that a variant is kind of similar to a CSS class. Only it’s not cascading, and you can’t apply many of these, and so on.</p> <p>A variant lets you swap a component’s style entirely, and to key into a theme style. So you can switch themes completely, each having their own set of variants, or you can switch variants:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-keyword z-operator z-spread z-ts">...</span> </span><span class="z-source z-ts"><span class="z-entity z-name z-label z-ts">variants</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">button</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-accessor z-ts">.</span><span class="z-punctuation z-accessor z-ts">.</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p>Variants can be generic, as in above, or can be component specific such as:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-keyword z-operator z-spread z-ts">...</span> </span><span class="z-source z-ts"><span class="z-entity z-name z-label z-ts">buttons</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">primary</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-accessor z-ts">.</span><span class="z-punctuation z-accessor z-ts">.</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p>And use it like so:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">button</span> <span class="z-variable z-other z-readwrite z-ts">variant</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>primary<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></code></pre> <p>And the piece of infrastructure that’s supposed to figure all this out, context, variant mix-in and theming is your UI framework.</p> <h2 id="themeprovider">ThemeProvider</h2> <p>Having that we’ve established that a theme spec is one main aspect of what we’re doing here, almost every next-gen UI framework has a <code>ThemeProvider</code> or uses the underlying infrastructure's <code>ThemeProvider</code> such as Emotion's or styled-component's.</p> <p>Basically it’s:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">theme</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>./theme<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">Index</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-cast z-expr z-ts"> <span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">ThemeProvider</span> <span class="z-entity z-name z-type z-ts">theme</span>=<span class="z-meta z-object z-type z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-field z-declaration z-ts"><span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">theme</span></span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">ThemeProvider</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span></span> </span></code></pre> <h2 id="box">Box</h2> <p>At a first glance <code>Box</code> looks like a simplistic component. However, what it holds is the entire infrastructure for figuring out themes, style props, variants and more. The idea is that there is a single smart component which you build on, and out of which all of your other components will derive.</p> <p>So for example to create a button you would do:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">Box</span> <span class="z-variable z-other z-readwrite z-ts">as</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>button<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span> <span class="z-keyword z-operator z-spread z-ts">...</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></code></pre> <p>And of course, when we say <code>Box</code> we're missing just one more word to make a winning combo: <code>Flex</code>. And so, may of the newer UI frameworks adopted both <code>Box</code> to signify the main building block that has all of the magic in it, and <code>Flex</code>, your main layout component to design your UI with.</p> <h2 id="concerns">Concerns</h2> <p><code>Box</code> contains multiple concerns, each takes care of a different aspect of a proper design system. Here's a breakdown (from <code>theme-ui</code>):</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-keyword z-control z-export z-ts">export</span> <span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">Box</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">styled</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>div<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">shouldForwardProp</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">(</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">boxSizing</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>border-box<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">margin</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">0</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">minWidth</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">0</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">base</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">variant</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">space</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">color</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">sx</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">props</span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span><span class="z-variable z-other z-object z-ts">props</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">css</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-brace z-round z-ts">)</span></span> </span></code></pre> <p>This means that <code>theme-ui</code>'s <code>Box</code> components supports variants, spacing and space-aware props that connects these with our theme, colors -- which is very similar, and your magical <code>sx</code> and <code>css</code> props that we discussed previously.</p> <p>And of course it takes care of your go-to fixes such as stating that this is a <code>border-box</code> layout model.</p> <p>A good starting point to understand each UI framework is to start looking at the source for the <code>Box</code> component. Here is <a href="https://github.com/chakra-ui/chakra-ui/blob/master/packages/chakra-ui/src/Box/index.js">Chakra UI</a> <code>Box</code> component, which in this case includes concerns such as <code>shadow</code> and <code>typography</code>.</p> <h2 id="composition-vs-inheritance">Composition vs. Inheritance</h2> <p>We’ve been saying “concerns” a lot. If we look at React and the component approach from a bird’s eye view, we discover that we’re having two patterns hiding in plain sight: composition and inheritance.</p> <p>Inheritance is augmenting and creating new components by using existing ones, so for example:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">Button</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-string z-template z-ts"><span class="z-variable z-other z-object z-ts">styled</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-tagged-template z-ts">div</span></span><span class="z-string z-template z-ts"><span class="z-punctuation z-definition z-string z-template z-begin z-ts">`</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-string z-template z-ts">... </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-string z-template z-ts"><span class="z-punctuation z-definition z-string z-template z-end z-ts">`</span></span></span> </span></code></pre> <p>Here we take a <code>div</code> and "inherit" from it to create a <code>Button</code>.</p> <p>When we look at the Box concept, we discover something different, this is actually composition:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-keyword z-control z-export z-ts">export</span> <span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">Box</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">styled</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>div<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">shouldForwardProp</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">(</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">base</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">variant</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">space</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">color</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">sx</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">props</span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span><span class="z-variable z-other z-object z-ts">props</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">css</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-brace z-round z-ts">)</span></span> </span></code></pre> <p>The traits, or concerns, <code>base</code>, <code>variant</code> and so on, are creating multiple inheritance trees or more specifically -- a concerns/mixin based model, which we know from Software Engineering is more flexible, resilient and adaptable.</p> <p>So by opting into the new way to build design systems in React, you’re also opting into a better software engineering model.</p> <h2 id="toolkit-vs-framwork">Toolkit vs. Framwork</h2> <p>As my own opinionated view, I look at the following frameworks as toolkits, not frameworks. It’s hard to accept a toolkit, because we’ve been used to all-in-one frameworks back from Bootstrap, and up to Antd.</p> <p>But when you realize what I’ve discussed up until now, you understand that with composition and a good way to model a framework you can build you own, on the condition that you don’t really need a full-blown enterprise-ready UI frameworks with complex data tables, calendars, dropdowns and more.</p> <h2 id="rebass">Rebass</h2> <p><img src="https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/13.png" alt="" /></p> <p><a href="https://rebassjs.org/">Rebass</a> was one of the most popular frameworks that pushed this new approach into mass adoption. Approaching both mobile (React Native) and Web, it’s been a simple yet pragmatic framework for a long while.</p> <p>As we’ll see with <a href="https://github.com/jxnblk">Brent Jackson</a>, it evolved, struck different roots in the process, which mind you — is the better way to do things in software, into different and better frameworks.</p> <h2 id="styled-system">Styled System</h2> <p><img src="https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/14.png" alt="" /></p> <p>One of the infrastructure that we got from all that action was <a href="https://github.com/styled-system/styled-system">styled-system</a> which is the infrastructure under most of these new-gen UI frameworks.</p> <p>It’s actually easy to build your own framework using styled-system, as you get many of the <em>concerns</em> for free.</p> <h2 id="theme-ui">Theme UI</h2> <p><img src="https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/15.png" alt="" /></p> <p><a href="https://theme-ui.com/">Theme UI</a> is an evolution of work that Brent already did, you can read about the <a href="https://jxnblk.com/blog/themeability/">history here</a>. For a long while it had no components of its own, and so it was seen as a foundational framework, and in that sense people still preferred to use Rebass (also from Brent).</p> <p>Recently, Theme UI got a <a href="https://theme-ui.com/components">components kit</a> which makes it eligible for a one-stop-shop framework. I’ve personally moved to using Theme UI exclusively instead of combining it with something else, like Rebass.</p> <p>It introduces the <code>sx</code> prop, which is a way to combine themes and the <code>css</code> prop from Emotion.</p> <h2 id="chakra-ui">Chakra UI</h2> <p><img src="https://rng0.io/blog/4-new-theme-based-react-ui-toolkits-and-why-its-going-to-change-how-you-think/16.png" alt="" /></p> <p><a href="https://chakra-ui.com/">Chakra UI</a> is a polished, complete and friendly framework. It builds on top of every concept we’ve mentioned here, and still, tries to be compatible with Theme UI for a good degree.</p> <h2 id="styling-after-styling">Styling after Styling</h2> <p>So once you’ve picked your UI framework, how do you customize your styles? There are a few options.</p> <p>One, is use the <code>sx</code>, <code>css</code>, or style props to create your own variant of a component. For this, you'll have to create a new component:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">StyledButton</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">Button</span> <span class="z-variable z-other z-readwrite z-ts">sx</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>{<span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">background</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>dark<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span></span>}<span class="z-keyword z-operator z-relational z-ts">&gt;</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">Button</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></code></pre> <p>Another way, is to use forward refs and build a new component from an infrastructural point of view. In other words, it’s still just a <code>Box</code>. Here's how Chakra does it:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">Badge</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">forwardRef</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-meta z-parameter z-object-binding-pattern z-ts"><span class="z-punctuation z-definition z-binding-pattern z-object z-ts">{</span> <span class="z-variable z-parameter z-ts">variantColor</span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>gray<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-parameter z-ts">variant</span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>subtle<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-keyword z-operator z-rest z-ts">...</span><span class="z-variable z-parameter z-ts">props</span> <span class="z-punctuation z-definition z-binding-pattern z-object z-ts">}</span></span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">ref</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">useVariantColorWarning</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>Badge<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">variantColor</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">badgeStyleProps</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">useBadgeStyle</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">color</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-readwrite z-ts">variantColor</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">variant</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-parameter z-ts">Box</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">ref</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">ref</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">display</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>inline-block<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-spread z-ts">...</span><span class="z-punctuation z-accessor z-ts">.</span> </span></span></span></span></code></pre> <p>Lastly, in the general sense you can still use <code>styled-components</code>, either the Emotion <code>styled</code> version or if for some reason you want to mix <code>styled-components</code> and Emotion (in reality it works but officially there are caveats):</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">Button</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">styled</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">Box</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-string z-template z-ts"><span class="z-punctuation z-definition z-string z-template z-begin z-ts">`</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-string z-template z-ts"> ... </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-string z-template z-ts"><span class="z-punctuation z-definition z-string z-template z-end z-ts">`</span></span></span> </span></code></pre> <p>If in doubt, since this entire theory that we’ve built in this article really creates less code and cruft, the best thing to do is to dive into the Theme UI codebase and see how new components are built.</p> <p>With this added infrastructure magic you’ll be surprised how elegantly you can create new components and experiences without creating a lot of noise.</p> <h2 id="finding-themes">Finding Themes</h2> <p>Since themes are the new hot thing here, you can now mix and match and find inspiration for styling in themes that already exist. Take a <a href="https://github.com/system-ui/theme-ui/tree/master/packages">look here</a> at Theme UI’s packages list, and see how themes are made.</p> <p>Not only that, if you’re a fan of <a href="https://kyleamathews.github.io/typography.js/">Typography</a> you have a <code>toTheme</code> helper that can convert a Typography theme to be a first-class citizen Theme UI theme.</p> <h2 id="conclusion">Conclusion</h2> <p>I believe the above theory, discussion, building blocks, and implementation and showcases are the next step in UI frameworks, much thanks to the very observant and skillful <a href="https://github.com/jxnblk">Brent Jackson</a>. I also believe that in software engineering, the healthy way sometimes is to build something, and then <em>start from scratch</em> a couple times to get at something perfect, which is exactly what was done in the case of the frameworks we’ve seen.</p> Building Admission Controllers for Kubernetes with Lightscreen Fri, 24 Jan 2020 00:00:00 +0000 Unknown https://rng0.io/blog/building-admission-controllers-for-kubernetes-with-lightscreen/ https://rng0.io/blog/building-admission-controllers-for-kubernetes-with-lightscreen/ <p><img src="https://rng0.io/blog/building-admission-controllers-for-kubernetes-with-lightscreen/1.png" alt="" /></p> <p><a href="https://github.com/SpectralOps/lightscreen">Lightscreen</a> is one component out of the <a href="https://spectralops.io">Spectral</a> platform for Kubernetes. It’s a modular <a href="https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/">admission controller</a> toolkit for Kubernetes built in Go for Go developers.</p> <h3 id="a-word-about-admission-controllers">A Word About Admission Controllers</h3> <p><img src="https://rng0.io/blog/building-admission-controllers-for-kubernetes-with-lightscreen/2.png" alt="" /></p> <p>The concept of admission is just one way to model how we allow or forbid objects into our cluster.</p> <p>Kubernetes defines a pipeline of controllers that are active on <code>kube-apiserver</code> and that can participate in admission, a few of them are enabled by default such as:</p> <ul> <li>MutatingAdmissionWebhook</li> <li>ValidatingAdmissionWebhook</li> </ul> <p>Which means you can create your own service that offers a webhook to participate in the admission decision flow. You can add any policy that you think is valuable for your cluster, that is not pre-baked into Kubernetes already, or is too complicated or too domain-specific to be included.</p> <p>In the context of <img src="https://rng0.io/blog/building-admission-controllers-for-kubernetes-with-lightscreen/3.png" alt="Spectral" /></p> <p>Spectral uses <a href="https://github.com/spectralops/lightscreen">Lightscreen</a> as a modular platform for creating admission webhooks to allow or forbid an image that has not been scanned and approved by Spectral.</p> <p>We must note that the design of admission within Kubernetes has changed quite a few times which was a bit alarming to get around (breaking APIs, flawed implementation in the case of bypassing parts of hooks with a mutating hook, and other interesting issues). However at this stage we feel that the current Kubernetes APIs are stable enough to build on, and so we did.</p> <h2 id="building-your-own-image-notary-for-kubernetes">Building Your Own Image Notary For Kubernetes</h2> <p><img src="https://rng0.io/blog/building-admission-controllers-for-kubernetes-with-lightscreen/4.png" alt="" /></p> <p>Unlike <a href="https://www.openpolicyagent.org/">opa</a>, Lightscreen encourages to build your rules, logic and workflow in Go, and produce a self-contained static binary to deploy as an admission webhook service onto your cluster.</p> <p>Let’s look at how to build your own image notary with Lightscreen. The full example exists on our <a href="https://github.com/spectralops/lightscreen">Github repo</a> if you prefer to follow by reading code.</p> <p>In our context for this example story, the notary will look at an incoming request, get each image name, figure out its relevant SHA signature, compare it to an existing known one from an approved signatures list, and either admit or reject the request based on that.</p> <p>First, creating a new server has just a single starting point that’s provided by Lightscreen:</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">main</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-operator z-variadic z-go">...</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">server</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">admission</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">NewServer</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">admission</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">ServerOptions</span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">Config</span><span class="z-punctuation z-separator z-go">:</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">config</span><span class="z-punctuation z-separator z-go">,</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">Development</span><span class="z-punctuation z-separator z-go">:</span> <span class="z-variable z-other z-go">development</span><span class="z-punctuation z-separator z-go">,</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">Address</span><span class="z-punctuation z-separator z-go">:</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">address</span><span class="z-punctuation z-separator z-go">,</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">logger</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <p>We packages all of the machinery of being a first-class Kubernetes admission controller into a nice <code>Server</code> abstraction, that expects either a <code>Mutation</code> action or a <code>Validation</code> action to be added later.</p> <p>Let’s look at the currently configured actions, both mutation and validation actions:</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">main</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-operator z-variadic z-go">...</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">server</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">admission</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">NewServer</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">admission</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">ServerOptions</span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">Config</span><span class="z-punctuation z-separator z-go">:</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">config</span><span class="z-punctuation z-separator z-go">,</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">Development</span><span class="z-punctuation z-separator z-go">:</span> <span class="z-variable z-other z-go">development</span><span class="z-punctuation z-separator z-go">,</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">Address</span><span class="z-punctuation z-separator z-go">:</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">address</span><span class="z-punctuation z-separator z-go">,</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">logger</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">logger</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Infof</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>mutations=<span class="z-constant z-other z-placeholder z-go">%v</span> validations=<span class="z-constant z-other z-placeholder z-go">%v</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">server</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">Actions</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">MutationMap</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">server</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">Actions</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">ValidationMap</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <p>By default, Lightscreen comes with preconfigured SHA resolver and validator actions.</p> <p>Each action is stored into an action map, and keyed by a unique identifier, so that even if actions are included by default, as long as you don’t use them, they’re not coming into effect.</p> <p>These actions are configured with a by-convention workflow configuration file — the main and only configuration for Lightscreen. Here’s an example:</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">mutations</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">type</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">resolve_sha</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">finder</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">crane</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">validations</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">type</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">admit_sha</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">admit</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-quoted z-double z-yaml"><span class="z-punctuation z-definition z-string z-begin z-yaml">&quot;</span>library/nginx@sha256:a8517b1d89209c88eeb48709bc06d706c261062813720a352a8e4f8d96635d9d<span class="z-punctuation z-definition z-string z-end z-yaml">&quot;</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-constant z-language z-boolean z-yaml">true</span> </span></code></pre> <p>And finally, serve:</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">main</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-operator z-variadic z-go">...</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">server</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">admission</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">NewServer</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">admission</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">ServerOptions</span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">Config</span><span class="z-punctuation z-separator z-go">:</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">config</span><span class="z-punctuation z-separator z-go">,</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">Development</span><span class="z-punctuation z-separator z-go">:</span> <span class="z-variable z-other z-go">development</span><span class="z-punctuation z-separator z-go">,</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">Address</span><span class="z-punctuation z-separator z-go">:</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">address</span><span class="z-punctuation z-separator z-go">,</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">logger</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-comment z-line z-go"><span class="z-punctuation z-definition z-comment z-go">//</span>server.Actions. </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">logger</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Infof</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>mutations=<span class="z-constant z-other z-placeholder z-go">%v</span> validations=<span class="z-constant z-other z-placeholder z-go">%v</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">server</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">Actions</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">MutationMap</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">server</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">Actions</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">ValidationMap</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">logger</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Infow</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>running<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>address<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">address</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>config<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">config</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>development<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">development</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">server</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Serve</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <p>Lightscreen will run as a Kubernetes compliant admission webhook controller.</p> <p>If you want to add or remove validation or mutation actions, you can explore the <code>.Actions</code> API:</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-variable z-declaration z-go">yourAction</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-function z-go">NewAction</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span><span class="z-source z-go"><span class="z-variable z-other z-go">server</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">Actions</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Add</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">yourAction</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></code></pre> <h2 id="creating-your-own-actions">Creating Your Own Actions</h2> <p><img src="https://rng0.io/blog/building-admission-controllers-for-kubernetes-with-lightscreen/5.png" alt="" /></p> <p>Creating your own actions is done by building them in Go. There are a couple abstractions that you use in order to build either a mutation or validation actions.</p> <p>Mainly, each action type should conform to the Action interface:</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-storage z-type z-keyword z-type z-go">type</span> <span class="z-entity z-name z-type z-go">Action</span> <span class="z-storage z-type z-keyword z-interface z-go">interface</span> <span class="z-meta z-type z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-type z-go"> <span class="z-entity z-name z-function z-go">Name</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span> </span></span><span class="z-source z-go"><span class="z-meta z-type z-go"> <span class="z-entity z-name z-function z-go">Run</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">context</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Context</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">unstructured</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Unstructured</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">error</span></span> </span></span><span class="z-source z-go"><span class="z-meta z-type z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <p>Where <code>unstructured</code> is a way Kubernetes internally represents <a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured">unstructured requests</a>. With <code>unstructured</code> you'll be able to probe a request, modify and reason about it in your Mutation or Validation actions.</p> <p>Here’s a mutating action that resolves an image name to the latest SHA (error handling removed for brevity):</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-meta z-function z-declaration z-go"><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-parameter z-go">r</span> <span class="z-keyword z-operator z-go">*</span><span class="z-storage z-type z-go">ResolveSHAAction</span><span class="z-punctuation z-section z-parens z-end z-go">)</span></span><span class="z-meta z-function z-declaration z-go"> <span class="z-entity z-name z-function z-go">Run</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-parameter z-go">ctx</span> <span class="z-variable z-other z-go">context</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Context</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-parameter z-go">p</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">unstructured</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Unstructured</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">error</span></span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">o</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-language z-blank z-go">_</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">gabs</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Consume</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">p</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">Object</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">containers</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">o</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Path</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>spec.containers<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Data</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-storage z-type z-keyword z-interface z-go">interface</span><span class="z-meta z-type z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span><span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">for</span> <span class="z-variable z-language z-blank z-go">_</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">c</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-keyword z-other z-go">range</span> <span class="z-variable z-other z-go">containers</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">container</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-language z-blank z-go">_</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">gabs</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Consume</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">c</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">img</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">container</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Path</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>image<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Data</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">resolved</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-language z-blank z-go">_</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">r</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">resolver</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Resolve</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">img</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">container</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Set</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">resolved</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>image<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-constant z-language z-go">nil</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <p>This will mutate the request and place an appropriate SHA in the <code>containers</code> section.</p> <p>The way Lightscreen works is that you build your own collection of Go-based mutation or validation actions and you mix and match these, producing as many self-contained binaries for your various controllers that you like.</p> <h2 id="why-build-rules-in-go">Why Build Rules In Go?</h2> <p>The fact that you build Lightscreen actions in Go is one of the main differences between Lightscreen and OPA, and it is the <em>only</em> way you can build new rules.</p> <p>The reasons we chose Go here, which is not something that’s natural for a “rule based” engine, are:</p> <ul> <li><em>Familiarity</em>: Familiar language and no learning curve</li> <li><em>Tooling</em>: you can test, observe, trace, profile rules in isolation, the same way you do Go code</li> <li><em>Reuse</em>: you can use your own codebase, utilities, infra, and domain</li> <li><em>Safety and performance</em>: you get the same guaranties as with Go, which you’re already familiar with. For example, say you want to perform many parallel requests within a validation action.</li> <li><em>Self contained</em>: there’s nothing like shipping a single, self-contained binary</li> <li><em>Many ways to package</em>: you can use parts of Lightscreen, you can use it as a starting-point for something greater, and many other ways to slice things</li> <li><em>YAGNI</em>: you ain’t gonna need it. If you know exactly what you want to build in terms of your admission workflow, and it’s not going to change much, you want something that gets out of your way and lets you code and ship a working service</li> </ul> <h2 id="what-s-next">What’s Next?</h2> <p>We use a version of Lightscreen internally in the Spectral platform. Every now and then we push a latest variant into the open source repo, which is being published <a href="https://github.com/spectralops/lightscreen">on Github</a> (being a security product, we use a private-public model for security reasons).</p> <p>In our <a href="https://github.com/spectralops/lightscreen">Lightscreen</a> repo there are a few more examples waiting for you, as well as documentation and code to try out.</p> Effective Work: Secrets To Working Effectively in the Age of Information Overload Mon, 13 Jan 2020 00:00:00 +0000 Unknown https://rng0.io/blog/effective-work-secrets-to-working-effectively-in-the-age-of-information-overload/ https://rng0.io/blog/effective-work-secrets-to-working-effectively-in-the-age-of-information-overload/ <p>I’ve just finished my book, working on it on and off in the span of the last 3 years, I finally had the chance to invest some serious time to gather all of the notes, half-baked texts, drafts, and go through the painful process of producing something that I can be proud of.Here are a few excerpts of the chapter titled: <em>“Flow”</em></p> <h3 id="low-energy">Low Energy</h3> <p>How effective you are with your work and tasks depends on your mental state. Ever wake up for an alert being on-call and tried to troubleshoot an incident? It’s one of those memories that tend to stick. Can you think clearly and take on “heavy” tasks when you’re feeling under the weather? Do you feel like making an impact on the world when you’re plain tired?</p> <p>On the flip side, there are many situations in which you want to tear up the world. You just got great feedback on a product you’re building, people shared your article and gave you great feedback: “keep on writing!”, or you’ve got lots of hours of sleep and feel super energized, or you just finished a fantastic workout at the gym.</p> <p>Understanding these mental states which have a close relationship to “effectiveness states”, and being aware of them is hard. You might be just starting to get a cold, not noticing it, and hitting a brick wall with an algorithm you’re developing and you have no idea why something trivial became so complex to you at that moment. You can’t understand why nothing is working, you’re quirky, angry, and frustrated — and it doesn’t help.</p> <h3 id="identify-one-mental-state">Identify One Mental State</h3> <p>If there were just one mental state that you can train yourself to identify and be aware of, I’d say that would be — identify the times when you have low amounts of energy. When you have low amounts of energy, all you need to do is pick a different kind of task. This also means you want to classify a special category of tasks as “low energy” (you can use a label or what have you). Things like “tidy up my drawers”, or “order groceries” or “call the cable company” are useful, but obviously you don’t want to use your premium high-energy time for these kind of tasks. Use the “low quality” time where you have low energy.</p> <p>This also means we discovered a principle: time is not of constant and equal quality. Or in other words an hour isn’t always an hour.</p> <p>As for the rest of the various ups and downs of our effectiveness states — being sick, being moody, and so on, it’s very helpful to be able to understand and listen to your body (look up: mindfulness), but I have to say there’s very little you can actually *do* at that moment to be mark yourself “effective”. When you’re sick you need to rest, as the doctor ordered.</p> <h3 id="the-circadian-rhythm">The Circadian Rhythm</h3> <p>There are some things that are common to all of us, in theory. In theory, we all should be following the <a href="https://en.wikipedia.org/wiki/Circadian_rhythm">Circadian Rhythm</a> to varying degrees. This means that as humans our bodies and minds were programmed, like all animals, around a cycle that’s rooted in our sleep and waking time. The Circadian Rhythm tells us, for example that we’re at our highest alertness at 10am. Guess what we’re all doing at 10am? that’s right. Stand up meetings. What a waste.</p> <p>A clever manager will move stand up meetings to 14:30pm, where we have “best coordination”. But that’s not the end of the story; I believe this rhythm is just a starting point, as we’re all also affected by things that are new to the world (relatively speaking) such as flights, substances we consume (caffeine, alcohol), and things like blue light from screens which trick our minds about the conditions outside (light vs darkness) and mess with our sleep cycle. The core idea is that you’re human: listen to yourself, and try to come up with a way to measure how *you* react to the Circadian Rhythm.</p> <p>For example, everyone says going to the Gym first thing in the morning is the best. I get absolutely dizzy, low energy, bad gym sessions in the mornings. However I’m completely pumped for planning and prioritization work first thing in the morning. So instead, I use mornings to plan and work; nights for sports and gym. Just to note that it took me a few years to accept this as my routine, since everyone kept telling me Gym in the morning is the best; this way I learned there’s no “best” when it comes to *your* fitness habits, there’s only *you* and the ways you have to measure if things are going well for you.</p> <h3 id="know-how-to-stop">Know How to Stop</h3> <p>Here’s a trick I use often when I write code. Contrary to what people might expect, I stop my work and make an effort to break my code. Sometimes I just either write out — right there and there — in plain english on <em>code</em> — what I need to do to continue such as “make sure to reimplement the file reader” which spits out a ton of compiler errors, or the more civilized version of this: write a failing test.</p> <p>The idea is that the next morning or evening when I open up my project to continue, I start in a “hot seat”. I have something that I have to fix immediately in order to start working. Often the act of fixing this *simple* broken code puts me in the right mental state, which is where I stopped the evening or morning before.</p> <p>So the idea is to leave yourself some mental breadcrumbs. In code it is: break your build in a simple way, you don’t want something fancy because you don’t want to be stuck <em>really</em> figuring out how to fix a broken build.</p> <p>There’s just one problem — if you’re going away for a long time, say more than a couple of days from this project, make sure you do the complete opposite. Tidy up everything that you can, make sure the project is in pristine condition before you leave it; you can’t afford coming back to a broken project when you’re completely out of context because you’ve been away for a week or two.</p> <p>I usually tidy up everything and maintain what I call a “work log” on Evernote where I just scribble the day’s happenings, like an informal journal, in a tone that sounds like speaking to a friend. This is an investment — it takes energy to maintain this kind of log so I only do that when I know that I’m context switching between projects for an interval of weeks or a good number of days. A couple weeks later I can read this log and rebound to a great amount of detail.</p> <h2 id="commitment-anxiety">Commitment Anxiety</h2> <p>Guess what happens when you have everything planned out? Well, you still haven’t solved the problem of finding more time, you just have things planned out.</p> <p>Sometimes when you do your homework right and you plan everything to the letter, you rediscover that time is scarce. You have your 3 hours every evening, and you have your roadmap all laid out, but you find out that you have a task that you’ve estimated to be 1.5 hours — but you never get to it.</p> <p>You’re not sure if you can lift that kind of a heavy weight; and that investing 1.5 hours — half of your evening time in one bulk is going to give you the feeling you’re looking for: “I did a lot today”. Some times approaching this task is just a form of anxiety because it’s complicated or because it’s a lot to do. I call this task commitment anxiety.</p> <p>This commitment anxiety is fuelled by two factors: realizing you’re committing a good chunk of your allotment of time, naturally we stay away from this because we feel that we’re “paying” a lot of time into just one thing. The second factor is just our natural tendency to conserve energy, we try to keep a balance of not spending energy (which is why we procrastinate).</p> <h3 id="take-a-nibble">Take a Nibble</h3> <p>Here’s how I break this anxiety: take a nibble. For every task that weights above an hour, if you have a time window that allows for starting it — start just 10 minutes of work out of this task. After ten minutes, one of two things will happen:</p> <ol> <li>You’ll just be pulled in and execute this task, overcoming commitment anxiety, or</li> <li>You’re going to realize this task needs to be split into two or more subtasks, the first part being “research” or “POC” and the second part being just the boring and dirty work of building something that’s already spelled out to the letter.</li> </ol> <p>Basically, <em>you want to take away choices</em>, which means your commitment anxiety was simply pronouncing the risk in a task — something that you’re not aware of but deep inside you *know* a task contains a good deal of unknowns and your brain wants to stay away from it to avoid failure or the unexpected. You can make the 10 minutes “nibble” work for you while you commute in the morning, before sleep or at another casual point in your day so that it’s no big deal to start.</p> <p>As long as you understand the mechanics of this kind of “anxiety”, you might come up with a different technique to “jumpstart” it. The principle is this: you need to learn how to start things which feels uncomfortable starting. Be great at starting, the rest will follow.</p> Why You Should Canary Test In Production Sat, 15 Jun 2019 00:00:00 +0000 Unknown https://rng0.io/blog/why-you-should-canary-test-in-production/ https://rng0.io/blog/why-you-should-canary-test-in-production/ <p><img src="https://rng0.io/blog/why-you-should-canary-test-in-production/1.jpg" alt="" /></p> <p>Testing, test strategies and tooling change often. Eight years ago we were all drinking the <a href="https://en.wikipedia.org/wiki/Behavior-driven_development">BDD coolaid</a>, dreaming about our Product Managers writing <a href="https://cucumber.io/">our specs in Cucumber</a> while we just happily write the code that makes them pass.</p> <p>It took the testing world by a storm. But then it stopped.</p> <p><img src="https://rng0.io/blog/why-you-should-canary-test-in-production/2.png" alt="" /></p> <p>The pendulum has swung, and today we’re seeing more of the simplistic style of testing taking the stage. Go’s <a href="https://golang.org/pkg/testing/">testing package</a> is one notable example of mainstream, simple, testing approaches. Some people even <a href="https://github.com/substack/tape">wanted a simpler test output</a>.</p> <p><img src="https://rng0.io/blog/why-you-should-canary-test-in-production/3.png" alt="" /></p> <p><a href="https://jestjs.io/">Jest</a> for example, is a mix. It takes a simplistic BDD-style testing approach; it takes BDD-the-language but not the bells and whistles of the frameworks, which leaves the extreme idealistic approaches out. It even has a minimalistic set of matchers, as opposed to the massive amount of <a href="https://github.com/thoughtbot/shoulda-matchers">matcher</a> libraries that we used to see come out every now and then.</p> <p>So, testing, test strategies, and frameworks all change as years go by.</p> <p>Even their backing technologies change.</p> <h2 id="contracts-are-king">Contracts are King</h2> <p>Contracts don’t <em>want</em> to change often. SLAs ( <a href="https://en.wikipedia.org/wiki/Service-level_agreement">Service Level Agreements</a>) and nonfunctional requirements might <em>never</em> change, because you’ve got a lawyer to sign an actual <em>courtroom-backed</em> contract specifying your uptime, your support policy, and your service quality.</p> <p>The result is that we have to test our contracts as if we’re our own clients — that means <a href="http://softwaretestingfundamentals.com/black-box-testing/">black box testing</a>.</p> <p>We can expect SLA definitions such as:</p> <ol> <li>%99.9 uptime (or any other specification in percent).</li> <li>All requests come below one (1) second for main flows, eight (8) seconds for secondary flows.</li> <li>Clean security postures on all APIs (i.e. don’t expose server headers).</li> <li>Early notification of breakage and maintenance, up to 72 hours ahead.</li> <li>And so on… (you get the point).</li> </ol> <p>We can translate these requirements to tests:</p> <ol> <li>Periodical test for a main endpoints (see if up or down, such as the <code>/login</code> endpoint).</li> <li>Benchmark public endpoint requests after deploys and periodically.</li> <li>Verify security standards on public endpoints.</li> <li>Test things that usually break — response contracts, error codes, SSL/TLS certificates.</li> </ol> <p>What tools do we have? well, we can use BDD with Cucumber to test APIs, like <a href="https://developer.ibm.com/recipes/tutorials/api-testing-framework-using-bdd-approach-through-cucumber-and-rest-assured/">in this article</a>:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">Scenario Outline: Check if user is able to submit GET API request </span><span class="z-text z-plain"> Given I want to execute service &lt;serviceName&gt; </span><span class="z-text z-plain"> When I submit the GET request as per the data in Excel Worksheet &lt;EWorksheet&gt; </span><span class="z-text z-plain"> And I validate status code </span><span class="z-text z-plain"> And I validate mandatory tag in response from Excel Worksheet &lt;EWorksheet&gt; </span><span class="z-text z-plain"> And I validate response content </span><span class="z-text z-plain"> And I validate tag values in response from Excel Worksheet &lt;EWorksheet&gt; </span><span class="z-text z-plain"> And I validate header parameter in response </span></code></pre> <p>This looks fancy. It’s declarative and it’s english. But if you read the rest of <a href="https://developer.ibm.com/recipes/tutorials/api-testing-framework-using-bdd-approach-through-cucumber-and-rest-assured/">the article</a>, before you know it, you’ll find out you have to fill in and implement an API testing framework yourself.</p> <p>But even if you did, there’s arguments to be had about how maintainable this look like for an API testing solution. <em>And</em>, you have to carry cucumber around every time you want to do this, possibly in projects and teams that don’t care for, or don’t have cucumber support.</p> <h2 id="fine-let-s-program-everything">Fine, Let’s Program Everything</h2> <p>You might say, if we’re building everything from scratch, why not get rid of cucumber, and just code our own framework?</p> <p>Here’s the idea using Jest:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">it</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>makes a request<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span></span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">request</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>http://example.com<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-meta z-brace z-round z-ts">)</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-promise z-ts">then</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">body</span></span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span></span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">expect</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">body</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">toMatchSnapshot</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <p>This would have been a reasonable approach. But there’s a ton of boilerplate in it — if I’m expecting the response body to be in a certain shape, isn’t <em>that</em> all I’m supposed to specify?</p> <h2 id="contract-driven-testing">Contract Driven Testing</h2> <p><img src="https://rng0.io/blog/why-you-should-canary-test-in-production/4.png" alt="" /></p> <p>Maybe what we’re really looking for is contract driven testing? A bunch of testing frameworks and contract specification formats:</p> <ul> <li><a href="https://swagger.io">Swagger</a></li> <li><a href="https://apiblueprint.org/">API blueprint</a></li> <li><a href="https://raml.org/">RAML</a></li> </ul> <p>All of these have tools to <em>validate</em> their own contract specifications, to varying levels of success. Tools like <a href="https://dredd.readthedocs.io">Dredd</a>, and <a href="https://apidevtools.org/swagger-cli/">Swagger CLI</a>.</p> <p>But what about actual content? nope. There’s partial support. Also because for example, in Swagger (and the OpenAPI spec) <em>examples</em> are kind of <a href="https://swagger.io/docs/specification/adding-examples/">secondary</a>; they’re an extension. For good reason — Open API is a contract specification format; a contract says nothing about actual API response content values, it specifies types and constraints.</p> <h2 id="trust-no-one">Trust No One</h2> <p>What about testing <em>your</em> dependencies? Say testing Twitter, if your product depends on that? How about some of your enterprise service partners?</p> <p>You might not be able to get the specs for their contract, and they might not even have one. But you certainly know how what response and SLAs you’re expecting from them.</p> <p>How do we test <em>that</em> with contract driven testing tools? (spoiler: we don’t).</p> <h2 id="running-fast-and-breaking-things">Running Fast And Breaking Things</h2> <p>Some times you run fast and break things. But some things cannot break.</p> <p>We usually take a bottom up approach — unit tests, integration tests, functional tests, and sanity tests.</p> <p>But when running fast, we deploy a lot. The match between:</p> <ol> <li>Deploy workflow, orchestration, and tools (relatively static).</li> <li>Your software (changes a lot).</li> </ol> <p>Is being exercised many times a day. When either change, and especially the deploy workflow changes, we don’t <em>really</em> have a mainstream way to test, or actually, meta-test all that. Does this even have a name?</p> <p>That’s a top-to-bottom approach — black box testing. And for matching your workflow and software — that’s called <a href="https://dzone.com/articles/canary-tests">Canary tests</a>; and we need a tool to write and execute these quickly and easily.</p> <h2 id="introducing-canary">Introducing: Canary</h2> <p><img src="https://rng0.io/blog/why-you-should-canary-test-in-production/5.png" alt="" /></p> <p><a href="https://canaryops.io">Canary</a> is the Swiss army-knife of canary testing. It lets you test:</p> <ul> <li>Contracts</li> <li>Content, body, headers, status</li> <li>Certificates (SSL/TLS)</li> <li>Benchmarks and performance</li> <li>Uptime and SLAs</li> </ul> <p>All in a generic, technology-independent way. Canary is written in <a href="https://www.rust-lang.org/">Rust</a> a modern systems programming language that is: fast, secure, and has an extremely low footprint.</p> <p>Here’s your first Canary test:</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">http_interactions</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"><span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">request</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">uri</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">https://google.com</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">response</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">status_code</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-quoted z-double z-yaml"><span class="z-punctuation z-definition z-string z-begin z-yaml">&quot;</span>200<span class="z-punctuation z-definition z-string z-end z-yaml">&quot;</span></span> </span></code></pre> <p>And here’s what happens when we execute it:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> canary run</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">✔</span></span><span class="z-meta z-function-call z-arguments z-shell"> verify: request - (https://google.com</span><span class="z-meta z-function-call z-shell"></span>) </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Ran</span></span><span class="z-meta z-function-call z-arguments z-shell"> 1 interactions in 246ms. Success: 1 Failure: 0 Skipped: 0</span> </span></code></pre> <p>You can try this <em>right now</em> with nothing but Docker on your machine:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> docker run<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>v</span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span><span class="z-meta z-group z-expansion z-command z-parens z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-parens z-begin z-shell">(</span><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">pwd</span></span><span class="z-punctuation z-section z-parens z-end z-shell">)</span></span><span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span>:/canary jondot/canary init</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> docker run<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>v</span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span><span class="z-meta z-group z-expansion z-command z-parens z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-parens z-begin z-shell">(</span><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">pwd</span></span><span class="z-punctuation z-section z-parens z-end z-shell">)</span></span><span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span>:/canary jondot/canary run</span> </span></code></pre> <p>Quick. Simple. Fast.</p> <h2 id="a-bag-of-opportunities">A Bag of Opportunities</h2> <p><img src="https://rng0.io/blog/why-you-should-canary-test-in-production/6.png" alt="" /></p> <p>Canary is built to accommodate many use cases at once:</p> <h3 id="your-own-project-and-repository-flow"><strong>Your own project and repository flow.</strong></h3> <p>Simply check in your <code>.canary.yaml</code> file, and include Canary in your CI. Here it is on <a href="https://travis-ci.org">Travis CI</a>:</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-comment z-line z-number-sign z-yaml"><span class="z-punctuation z-definition z-comment z-line z-number-sign z-yaml">#</span> ... </span></span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">script</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">yarn test</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">yarn deploy:staging</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">curl http://get.canaryops.io/install | sh</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">canary run &amp;&amp; yarn deploy:production</span> </span></code></pre> <p>Canary will fail the build if needed.</p> <h3 id="as-a-centralized-sla-validation-tool"><strong>As a centralized SLA validation tool</strong>.</h3> <p>If you have an engineering effectiveness or <a href="https://en.wikipedia.org/wiki/Site_Reliability_Engineering">SRE</a> team in your organization, they will see Canary as a great tool for validating healthy production standards.</p> <p>A central repo with benchmarks, validations, and cert checks is something Canary can handle well.</p> <p>Here’s how to generate various types of interactions:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> canary init <span class="z-keyword z-control z-regexp z-set z-begin z-shell">[</span>with<span class="z-keyword z-operator z-word z-shell">-</span>vars|certs|benchmark|default<span class="z-keyword z-control z-regexp z-set z-end z-shell">]</span></span> </span></code></pre> <h3 id="as-a-homebrew-global-testing-service"><strong>As a homebrew global testing service.</strong></h3> <p>Wrap Canary in a Lambda and have it execute from any point on the globe. Canary is built with Rust, and so startup time is extremely low, and its footprint is as compact as it gets.</p> <p>You can also use Canary’s various reporters to ship back JSON results to your aggregated logs service, for analytics, aggregation, and forensics. You’re in control.</p> <h2 id="take-canary-for-a-spin">Take Canary for a Spin</h2> <p>Canary is optimized for a quick and lightweight experience. There’s nothing to learn but using your engineering instincts.</p> <p>Check out <a href="https://canaryops.io/docs/getting-started">the documentation</a> and <a href="https://canaryops.io/features">the features</a>, as well as sign up to <a href="https://canaryops.io">get a full license for free</a></p> <p>For everything else, check out Canary’s official website: <a href="https://canaryops.io">https://canaryops.io</a></p> Formation: A Functional Middleware Infrastructure for Python Tue, 21 May 2019 00:00:00 +0000 Unknown https://rng0.io/blog/formation-a-functional-middleware-infrastructure-for-python/ https://rng0.io/blog/formation-a-functional-middleware-infrastructure-for-python/ <p><img src="https://rng0.io/blog/formation-a-functional-middleware-infrastructure-for-python/1.png" alt="" /></p> <p><a href="https://jondot.github.io/formation">Formation</a> is a generic functional middleware infrastructure for Python.</p> <p>With Formation, you can build production-grade software — whether you want to build resilient, circuit-breaker infused and operable HTTP clients, HTTP services, or easily apply best practices and standards to other kinds of software to be flexible, composable, and maintainable.</p> <p>Although <em>not tied to a web service framework</em>, it takes inspiration from Ruby’s Rack middleware, and Node’s connect. In the context of Python, you can think of it as a higher-level abstraction or “WSGI over anything”.</p> <h2 id="the-pipeline-abstraction-and-functions">The Pipeline Abstraction and Functions</h2> <p>A graphics <em>pipeline</em> in digital cameras often helps perform these (photography connoisseurs may notice it’s not a complete set nor an accurate set of steps, but it helps make a point):</p> <ol> <li>White balancing</li> <li>Color optimization</li> <li>Sharpening</li> <li>Cropping</li> </ol> <p><img src="https://rng0.io/blog/formation-a-functional-middleware-infrastructure-for-python/2.jpg" alt="" /></p> <p>White balancing is the act of finding how much <em>white</em> is in a picture, so that the camera can optimize for what we care about — like skin tone, over what we don’t care about — like sun glare from the sea.</p> <p>Before and after I shoot, I can select from several kinds of white balancing settings, regardless of the rest of the steps. I’m surgically changing one part, and all other parts including the pipeline itself keeps working.</p> <p>The idea of the pipeline abstraction is to organize and execute a set of similarily-behaving and interchangeable building blocks. This means:</p> <ol> <li>With a successful design, we can place different building blocks into our pipeline, as well as borrow building blocks from others. There should be nothing specific to our pipeline design or anyone else’s that limits that kind of reuse.</li> <li>We can compose, order, and enable or disable the building blocks, and the pipeline should allow for it to be easy and unsurprising.</li> <li>It should be trivial to create new blocks from scratch, without unnecessary ceremonies.</li> </ol> <p><img src="https://rng0.io/blog/formation-a-functional-middleware-infrastructure-for-python/3.png" alt="" /></p> <p>Here’s a makeshift pipeline with a traditional object-oriented design:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">blocks</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">balance</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">optimize</span></span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span> </span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">pipeline</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">img</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-statement z-loop z-for z-python"><span class="z-keyword z-control z-loop z-for z-python">for</span> <span class="z-meta z-generic-name z-python">block</span> <span class="z-keyword z-control z-loop z-for z-in z-python">in</span></span><span class="z-meta z-statement z-loop z-for z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">blocks</span></span></span><span class="z-meta z-statement z-loop z-for z-python"><span class="z-punctuation z-section z-block z-loop z-for z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">img</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">block</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">execute</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">img</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p>And here’s an example block which goes into that pipeline:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-class z-python"><span class="z-storage z-type z-class z-python"><span class="z-keyword z-declaration z-class z-python">class</span></span> <span class="z-entity z-name z-class z-python"><span class="z-meta z-generic-name z-python">Balance</span></span><span class="z-punctuation z-section z-class z-begin z-python">:</span></span> </span><span class="z-source z-python"><span class="z-meta z-function z-python"> <span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">execute</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">self</span><span class="z-punctuation z-separator z-parameters z-python">,</span> <span class="z-variable z-parameter z-python">img</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">img</span></span> </span></code></pre> <p>Those who identify this interface realize that it’s actually an interface for a function!. Here’s a simpler pipeline and blocks:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">balance</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">img</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-pass z-python">pass</span> </span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">optimize</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">img</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-pass z-python">pass</span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">blocks</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">balance</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">optimize</span></span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span> </span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">pipeline</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">img</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-statement z-loop z-for z-python"><span class="z-keyword z-control z-loop z-for z-python">for</span> <span class="z-meta z-generic-name z-python">f</span> <span class="z-keyword z-control z-loop z-for z-in z-python">in</span></span><span class="z-meta z-statement z-loop z-for z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">blocks</span></span></span><span class="z-meta z-statement z-loop z-for z-python"><span class="z-punctuation z-section z-block z-loop z-for z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">img</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">f</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">img</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p>What did we gain by moving away from a class (and by that, moving away from object-oriented design)? Well, we’ve got composability for free, because functions in programming, when pure, are much like functions in mathematics — composable:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">f</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">x</span></span><span class="z-keyword z-operator z-arithmetic z-python">*</span><span class="z-constant z-numeric z-integer z-decimal z-python">3</span> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">g</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">x</span></span><span class="z-keyword z-operator z-arithmetic z-python">*</span><span class="z-constant z-numeric z-integer z-decimal z-python">10</span> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">k</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">f</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">g</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">f</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">g</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">x</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p>Composing functions feel natural, and mechanical to code — boring code is great; it easily lends itself to abstraction and DRY.</p> <p>And with that in mind — here we compose with <a href="https://toolz.readthedocs.io/en/latest/api.html">toolz</a>, a functional programming library for Python:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">toolz</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">compose</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">optimizing_balancer</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">compose</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">blance</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">optimize</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p>But just to drive a point — back to object-oriented design and the function class — How do we compose a class with an <code>#execute</code> method? Well, we resort to design patterns.</p> <p>We break our first version of the execute interface by making each block accept another block, and we introduce state by having to store it:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-class z-python"><span class="z-storage z-type z-class z-python"><span class="z-keyword z-declaration z-class z-python">class</span></span> <span class="z-entity z-name z-class z-python"><span class="z-meta z-generic-name z-python">Balance</span></span><span class="z-punctuation z-section z-class z-begin z-python">:</span></span> </span><span class="z-source z-python"><span class="z-meta z-function z-python"> <span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-support z-function z-magic z-python">__init__</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">other_block</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-variable z-language z-python">self</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">other_block</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">other_block</span></span> </span><span class="z-source z-python"><span class="z-meta z-function z-python"> <span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">execute</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> logic for _actually_ composing things with other_block </span></span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-pass z-python">pass</span> </span></code></pre> <p>Any way we look at it, or any way we try to make sense out of this — the sad result is that we <em>patch</em> an unnatural abstraction <a href="https://en.wikipedia.org/wiki/Indirection">with more abstractions</a>, until it makes sense.</p> <h3 id="production-ready-services">Production Ready Services</h3> <p><img src="https://rng0.io/blog/formation-a-functional-middleware-infrastructure-for-python/4.png" alt="" /></p> <p>Wrapping our request logic with concerns</p> <p>Here are some common concerns for <strong>Web services</strong>(a representative set and by no means a covering one). You can go even further and claim these to be standard enforced on your company’s services.</p> <ul> <li><strong>Content caching and e-tags</strong> — before spilling content onto a TCP connection, digest it, cache it, or more generally perform a content-wise operation that helps the receiving end cache it and save bytes for other calls in the future.</li> <li><strong>Exception handling</strong> — when things blow up, we don’t want to send a stack trace or any parts of the internals of our service. We can always detect that, wrap it in something user-friendly, some troubleshooting advice, or retry advice.</li> <li><strong>Compression</strong> — it’s almost standard these days, with commodity computing power, to Gzip all content passing into and from services. In application servers like Apache and Nginx, this is a simple flag.</li> <li><strong>Throttling</strong> — It’s easy for a service to get overwhelmed these days, especially with the surge of serverless consumers with virtually endless scale and concurrency. What we do here is put inplace a module that blocks any consumer that have passed a certain threshold.</li> <li><em>Header aesthetics</em>_ — Security by obscurity, though not security, is still effective against certain kinds of attackers. If we don’t say we’re using Apache, an attacker might consider us not a good ROI and move to the next target. There are plenty of other “chatty” HTTP headers that reveal unnecessary information to any public consumer.</li> <li><strong>Circuit breaking</strong> — When a service is malfunctioning, it may be that it is down, or it operates in a degraded mode. To facilitate various levels of operative functions and fault tolerance, we use <a href="https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern">circuit breakers</a>. There are some variants for building these, and most famously there is <a href="https://github.com/Netflix/Hystrix">Hysterix</a>, by Netflix.</li> </ul> <h2 id="my-point-it-all-exists-for-clients-too">My Point? It All Exists For Clients, Too</h2> <p>We’ve discussed a Web service, or an API. We can push all these requirements to the client side — the consumer — and you’d be amazed to realize, they all are valid there as well.</p> <p>Historically, having Web services do all these hard work had facilitated a more shallow approach for clients — they don’t need to take care of those if the backend is.</p> <p>With the move to microservices the weights have been shifting — since there’s a service mesh that’s has complex inter-communication between services to perform a given request, it is now arguably as important to facilitate these on the client side as well as the server side.</p> <p>For example, if I clean my headers on my way out — I’m supporting better security. If I compress my content before it hits servers — I’m enabling faster, better traffic shapes. And lastly, if I avoid overwhelming a backend and stop short before I put bytes on the connection — I’m promoting a healthier backend server, of which the only overwhelming traffic event now becomes an exception rather than the norm.</p> <h2 id="middleware-a-blueprint">Middleware: A Blueprint</h2> <p><img src="https://rng0.io/blog/formation-a-functional-middleware-infrastructure-for-python/5.png" alt="" /></p> <p>These two stories bring us to our point: Middleware, functions, and client-side middleware. Here are a few examples for popular middleware libraries for web services and APIs:</p> <h3 id="python-s-wsgi"><strong>Python’s WSGI</strong></h3> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">application</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">environ</span><span class="z-punctuation z-separator z-parameters z-python">,</span> <span class="z-variable z-parameter z-python">start_response</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">start_response</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">200 OK<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-meta z-sequence z-tuple z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Content-Type<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">text/plain<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-sequence z-end z-python">)</span></span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-yield z-python">yield</span> <span class="z-storage z-type z-string z-python">b</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Hello, World<span class="z-constant z-character z-escape z-python">\n</span><span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span> </span></code></pre> <h3 id="ruby-s-rack"><strong>Ruby’s Rack</strong></h3> <pre data-lang="ruby" class="language-ruby z-code"><code class="language-ruby" data-lang="ruby"><span class="z-source z-ruby">app <span class="z-keyword z-operator z-assignment z-ruby">=</span> <span class="z-support z-class z-ruby">Proc</span><span class="z-punctuation z-accessor z-dot z-ruby">.</span><span class="z-keyword z-other z-special-method z-ruby">new</span> <span class="z-keyword z-control z-start-block z-ruby">do</span> <span class="z-meta z-block z-parameters z-ruby"><span class="z-punctuation z-definition z-parameters z-begin z-ruby">|</span></span><span class="z-meta z-block z-parameters z-ruby"><span class="z-variable z-parameter z-ruby">env</span><span class="z-meta z-block z-parameters z-ruby"><span class="z-punctuation z-definition z-parameters z-end z-ruby">|</span></span></span> </span><span class="z-source z-ruby"> <span class="z-punctuation z-section z-array z-ruby">[</span><span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-single z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&#39;</span>200<span class="z-punctuation z-definition z-string z-end z-ruby">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-ruby">,</span> <span class="z-punctuation z-section z-scope z-ruby">{</span><span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-single z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&#39;</span>Content-Type<span class="z-punctuation z-definition z-string z-end z-ruby">&#39;</span></span></span> <span class="z-punctuation z-separator z-key-value z-ruby">=&gt;</span> <span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-single z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&#39;</span>text/html<span class="z-punctuation z-definition z-string z-end z-ruby">&#39;</span></span></span><span class="z-punctuation z-section z-scope z-ruby">}</span><span class="z-punctuation z-separator z-sequence z-ruby">,</span> <span class="z-punctuation z-section z-array z-ruby">[</span><span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-single z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&#39;</span>A barebones rack app.<span class="z-punctuation z-definition z-string z-end z-ruby">&#39;</span></span></span><span class="z-punctuation z-section z-array z-ruby">]</span><span class="z-punctuation z-section z-array z-ruby">]</span> </span><span class="z-source z-ruby"><span class="z-keyword z-control z-ruby">end</span> </span></code></pre> <h3 id="node-s-connect"><strong>Node’s Connect</strong></h3> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">app</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">use</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function z-expression z-ts"><span class="z-storage z-type z-function z-ts">function</span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">req</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">res</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">res</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">end</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>Hello from Connect!<span class="z-constant z-character z-escape z-ts">\n</span><span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <p>All these work in a backend environment, serving as request handlers in a way. They all take and serve — with different kinds of interfaces — the same triplet; status code, headers, and body. This simple design goes back as far as <a href="https://en.wikipedia.org/wiki/Common_Gateway_Interface">CGI</a>.</p> <p>If we look at Node’s Connect, there’s already a nice <a href="https://www.npmjs.com/search?q=keywords:middleware">pile of middleware</a> out there for you to choose from. Every middleware you use is code you don’t write in your handlers, and is a standard you can set to let others know they don’t need to write that code either.</p> <h2 id="middleware-in-clients">Middleware in Clients</h2> <p>Let’s focus on HTTP clients for now. There’s already one popular example of a great HTTP client that supports middleware in Ruby: <a href="https://github.com/lostisland/faraday">Faraday</a>.</p> <p>With Ruby’s elaborate DSL capabilities, Faraday lets you compose new clients with middleware — which turns these into advanced and robust HTTP clients, and it almost looks like prose:</p> <pre data-lang="ruby" class="language-ruby z-code"><code class="language-ruby" data-lang="ruby"><span class="z-source z-ruby"><span class="z-support z-class z-ruby">Faraday</span><span class="z-punctuation z-accessor z-dot z-ruby">.</span><span class="z-keyword z-other z-special-method z-ruby">new</span><span class="z-punctuation z-definition z-group z-begin z-ruby">(</span><span class="z-keyword z-operator z-range z-ruby">...</span><span class="z-punctuation z-definition z-group z-end z-ruby">)</span> <span class="z-keyword z-control z-start-block z-ruby">do</span> <span class="z-meta z-block z-parameters z-ruby"><span class="z-punctuation z-definition z-parameters z-begin z-ruby">|</span></span><span class="z-meta z-block z-parameters z-ruby"><span class="z-variable z-parameter z-ruby">conn</span><span class="z-meta z-block z-parameters z-ruby"><span class="z-punctuation z-definition z-parameters z-end z-ruby">|</span></span></span> </span><span class="z-source z-ruby"> <span class="z-comment z-line z-number-sign z-ruby"><span class="z-punctuation z-definition z-comment z-ruby">#</span> POST/PUT params encoders: </span></span><span class="z-source z-ruby"> conn<span class="z-punctuation z-accessor z-dot z-ruby">.</span>request <span class="z-constant z-other z-symbol z-ruby"><span class="z-punctuation z-definition z-constant z-ruby">:</span>multipart</span> </span><span class="z-source z-ruby"> conn<span class="z-punctuation z-accessor z-dot z-ruby">.</span>request <span class="z-constant z-other z-symbol z-ruby"><span class="z-punctuation z-definition z-constant z-ruby">:</span>url_encoded</span> </span><span class="z-source z-ruby"> <span class="z-comment z-line z-number-sign z-ruby"><span class="z-punctuation z-definition z-comment z-ruby">#</span> Last middleware must be the adapter: </span></span><span class="z-source z-ruby"> conn<span class="z-punctuation z-accessor z-dot z-ruby">.</span>adapter <span class="z-constant z-other z-symbol z-ruby"><span class="z-punctuation z-definition z-constant z-ruby">:</span>net_http</span> </span><span class="z-source z-ruby"><span class="z-keyword z-control z-ruby">end</span> </span></code></pre> <p>There’s no such thing for Python — that is, until now.</p> <h2 id="introducing-formation">Introducing: Formation</h2> <p><a href="https://jondot.github.io/formation">Formation</a> is a generic middleware infrastructure for Python, as well as advanced HTTP client building infrastructure for Python — the same way Faraday makes all this possible in Ruby.</p> <p>Here’s a quick Formation HTTP client:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-annotation z-python"><span class="z-punctuation z-definition z-annotation z-python">@</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">formation</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-annotation z-python"><span class="z-meta z-generic-name z-python">client</span></span></span></span> </span><span class="z-source z-python"><span class="z-meta z-class z-python"><span class="z-storage z-type z-class z-python"><span class="z-keyword z-declaration z-class z-python">class</span></span> <span class="z-entity z-name z-class z-python"><span class="z-meta z-generic-name z-python">Google</span></span><span class="z-meta z-class z-inheritance z-python"><span class="z-punctuation z-section z-inheritance z-begin z-python">(</span></span></span><span class="z-meta z-class z-inheritance z-python"><span class="z-entity z-other z-inherited-class z-python">object</span><span class="z-punctuation z-section z-inheritance z-end z-python">)</span></span><span class="z-meta z-class z-python"><span class="z-punctuation z-section z-class z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">base_uri</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">https://google.com<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">middleware</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">request_logger</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">structlog</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">getLogger</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">response_as</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">html_response</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-function z-python"> <span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">search</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">self</span><span class="z-punctuation z-separator z-parameters z-python">,</span> <span class="z-variable z-parameter z-python">text</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-language z-python">self</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">request</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">get</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">/<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-variable z-parameter z-python">params</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">Query</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">text</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-statement z-conditional z-if z-python"><span class="z-keyword z-control z-conditional z-if z-python">if</span> <span class="z-meta z-qualified-name z-python"><span class="z-support z-variable z-magic z-python">__name__</span></span> <span class="z-keyword z-operator z-comparison z-python">==</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">__main__<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-block z-conditional z-if z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">google</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">Google</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-sequence z-tuple z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">xml</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">_code</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">_headers</span></span><span class="z-punctuation z-section z-sequence z-end z-python">)</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">google</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">search</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">larry page<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">xml</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">xpath</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">//title/text()<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p>And here’s a simple piece of code that we want to infuse with middleware magic, regardless of HTTP and clients:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">formation</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">wrap</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">requests</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">get</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">log</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">ctx</span><span class="z-punctuation z-separator z-parameters z-python">,</span> <span class="z-variable z-parameter z-python">call</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">started<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">ctx</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">call</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">ctx</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">ended<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">ctx</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">timeit</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">ctx</span><span class="z-punctuation z-separator z-parameters z-python">,</span> <span class="z-variable z-parameter z-python">call</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">started</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">now</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">ctx</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">call</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">ctx</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">ended</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">now</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-arithmetic z-python">-</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">started</span></span> </span><span class="z-source z-python"> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">ctx</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">duration<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">ended</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">ctx</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">to_requests</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">ctx</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">get</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">ctx</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">url<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">fancy_get</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">wrap</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">to_requests</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-variable z-parameter z-python">middleware</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">log</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">timeit</span></span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">fancy_get</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-section z-mapping z-begin z-python">{</span></span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">url<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">https://google.com<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-section z-mapping z-end z-python">}</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p>Lastly, here’s how we build a new middleware, just like that:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">log</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">ctx</span><span class="z-punctuation z-separator z-parameters z-python">,</span> <span class="z-variable z-parameter z-python">call</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">started<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">ctx</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">call</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">ctx</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">ended<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">ctx</span></span> </span></code></pre> <h2 id="middleware-should-be-easy">Middleware Should Be Easy</h2> <p>Middleware should be easy to add.</p> <p>What’s easy? a pure function, with no base class, no imports, no specialized programming language knowledge, and no prior knowledge about the host framework.</p> <p>Knowledge of a request — response interaction and the concept of context is needed; but that’s not a specialized form of knowlege in any way.</p> <p>Here is one middleware:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">log</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">ctx</span><span class="z-punctuation z-separator z-parameters z-python">,</span> <span class="z-variable z-parameter z-python">call</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">started<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">call</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">ctx</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p>A simple function that takes a <em>context</em> — can be anything, just a plain dict; and the next function it’s supposed to call — or in the case it decides to break the flow — not call.</p> <p>From this, we extract the entire middleware API in a single sentence:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">A middleware is a plain function that takes the current context and next middleware and is responsible to return a new context. </span></code></pre> <h3 id="decorators">Decorators</h3> <p>Formation is opinionated. The definition of “should be easy” rules out decorators; first, they are language-specific, and require prior knowledge, second, and more important they’re not flexible.</p> <p>Functions, given that they are first-class citizens in a language, inherently lend themselves to composition and high-level operations over functions.</p> <h3 id="first-class-operability">First Class Operability</h3> <p><img src="https://rng0.io/blog/formation-a-functional-middleware-infrastructure-for-python/6.png" alt="" /></p> <p>Here’s a definition for <em>Operability</em>:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">Operability is the ability to keep a piece of equipment, a system or a whole industrial installation in a safe and reliable functioning condition, according to pre-defined operational requirements. </span></code></pre> <p>In our more modest software world, I would define operability as:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">The ability of an organization to move a piece of production software from a malfunctioning to a functioning state as quickly, reliably and precisely as possible, accompanied by effective and precise learning about the malfunction. </span></code></pre> <p>A couple things Formation middleware does to support operable software are:</p> <ol> <li><code>context</code> -- the context middleware gather information about the running host that's needed to operate in case of malfunction. Information to answer questions like "from what Git version was this service deployed?", and "are these two threads running on the same process?"</li> </ol> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">context</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">namespace</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">service<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">scope</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">all<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">env</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">local<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">sha</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">dev<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">version</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">0.01<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">context_fn</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">get_context</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">getpid</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">os</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">getpid</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">gettid</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">thread</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">get_ident</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <ol> <li><code>logger</code> -- a <em>structured</em> request loggers packs all information into structured logs. Read <a href="http://www.structlog.org/en/stable/">here</a> about why structured logging promotes operable software much better than plain old text logs.</li> </ol> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">context_logger</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">structlog</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">getLogger</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p>And there’s more.</p> <h3 id="transparent-resilience">Transparent Resilience</h3> <p>Resilience in Formation comes in the form of two concepts:</p> <ol> <li><code>circuit_breaker</code> -- we don't want to bombard a service that's down, and bearly getting up because it's overwhelmed.</li> </ol> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">circuit_breaker</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"> </span></span><span class="z-source z-python"><span class="z-meta z-function z-parameters z-python"> <span class="z-variable z-parameter z-python">logger</span><span class="z-punctuation z-separator z-parameters z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function z-parameters z-python"> <span class="z-variable z-parameter z-python">name</span><span class="z-punctuation z-separator z-parameters z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function z-parameters z-python"> <span class="z-variable z-parameter z-python">fail_max</span></span><span class="z-meta z-function z-parameters z-default-value z-python"><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-numeric z-integer z-decimal z-python">5</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-separator z-parameters z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function z-parameters z-python"> <span class="z-variable z-parameter z-python">reset_timeout</span></span><span class="z-meta z-function z-parameters z-default-value z-python"><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-numeric z-integer z-decimal z-python">60</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-separator z-parameters z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function z-parameters z-python"> <span class="z-variable z-parameter z-python">state_storage</span></span><span class="z-meta z-function z-parameters z-default-value z-python"><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-language z-python">None</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-separator z-parameters z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function z-parameters z-python"> <span class="z-variable z-parameter z-python">exclude</span></span><span class="z-meta z-function z-parameters z-default-value z-python"><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-sequence z-list z-empty z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function z-parameters z-default-value z-python"></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"> </span></span></code></pre> <ol> <li><code>retry</code> -- in the case of a failed request, transparently retry it, without bothering the userland code. With flaky network, you don't even notice Formation is doing the retries for you.</li> </ol> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">retry</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">max_retries</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-numeric z-integer z-decimal z-python">3</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <h3 id="built-in-content-handling">Built In Content Handling</h3> <p>Formation can create a client that only understand a certain content type, to ease on your programming experience. For example, if an API supports only JSON (a reasonable thing these days), you don’t need that content sniffing code to understand what is it that you’ve just got in the response body, nor must you have to deal with manually parsing text into JSON.</p> <p>Here’s an example client, in which we explicitly choose <code>json_response</code> as what this client receives:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">formation</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">for_requests</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">client</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">json_response</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-annotation z-python"><span class="z-punctuation z-definition z-annotation z-python">@</span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-annotation z-python"><span class="z-meta z-generic-name z-python">client</span></span></span></span> </span><span class="z-source z-python"><span class="z-meta z-class z-python"><span class="z-storage z-type z-class z-python"><span class="z-keyword z-declaration z-class z-python">class</span></span> <span class="z-entity z-name z-class z-python"><span class="z-meta z-generic-name z-python">HttpBin</span></span><span class="z-meta z-class z-inheritance z-python"><span class="z-punctuation z-section z-inheritance z-begin z-python">(</span></span></span><span class="z-meta z-class z-inheritance z-python"><span class="z-entity z-other z-inherited-class z-python">object</span><span class="z-punctuation z-section z-inheritance z-end z-python">)</span></span><span class="z-meta z-class z-python"><span class="z-punctuation z-section z-class z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">response_as</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">json_response</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">base_uri</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">https://httpbin.org<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">middleware</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span> </span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">timeout</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-constant z-numeric z-float z-decimal z-python">0<span class="z-punctuation z-separator z-decimal z-python">.</span>1</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">accept</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">application/json<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">ua</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">the-fabricator/1.0.0<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">request_logger</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">structlog</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">getLogger</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-punctuation z-section z-sequence z-end z-python">]</span></span> </span></code></pre> <h3 id="structured-queries-for-developer-happiness">Structured Queries For Developer Happiness</h3> <p>Formation understands <a href="http://www.attrs.org/en/stable/">attrs</a> the most popular data-class-like library for Python. With Formation’s sister project <a href="https://github.com/jondot/attrs-serde">attrs-serde</a> it is possible to have fully structured queries in Formation clients, like so:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-annotation z-python"><span class="z-punctuation z-definition z-annotation z-python">@</span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-annotation z-python"><span class="z-meta z-generic-name z-python">serde</span></span></span></span> </span><span class="z-source z-python"><span class="z-meta z-annotation z-python"><span class="z-punctuation z-definition z-annotation z-python">@</span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-annotation z-python"><span class="z-meta z-generic-name z-python">attrs</span></span></span></span> </span><span class="z-source z-python"><span class="z-meta z-class z-python"><span class="z-storage z-type z-class z-python"><span class="z-keyword z-declaration z-class z-python">class</span></span> <span class="z-entity z-name z-class z-python"><span class="z-meta z-generic-name z-python">Query</span></span><span class="z-meta z-class z-inheritance z-python"><span class="z-punctuation z-section z-inheritance z-begin z-python">(</span></span></span><span class="z-meta z-class z-inheritance z-python"><span class="z-entity z-other z-inherited-class z-python">object</span><span class="z-punctuation z-section z-inheritance z-end z-python">)</span></span><span class="z-meta z-class z-python"><span class="z-punctuation z-section z-class z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">query</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">attrib</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-variable z-parameter z-python">metadata</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-section z-mapping z-begin z-python">{</span></span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">to<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">q<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-section z-mapping z-end z-python">}</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-annotation z-python"><span class="z-punctuation z-definition z-annotation z-python">@</span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-annotation z-python"><span class="z-meta z-generic-name z-python">client</span></span></span></span> </span><span class="z-source z-python"><span class="z-meta z-class z-python"><span class="z-storage z-type z-class z-python"><span class="z-keyword z-declaration z-class z-python">class</span></span> <span class="z-entity z-name z-class z-python"><span class="z-meta z-generic-name z-python">Google</span></span><span class="z-meta z-class z-inheritance z-python"><span class="z-punctuation z-section z-inheritance z-begin z-python">(</span></span></span><span class="z-meta z-class z-inheritance z-python"><span class="z-entity z-other z-inherited-class z-python">object</span><span class="z-punctuation z-section z-inheritance z-end z-python">)</span></span><span class="z-meta z-class z-python"><span class="z-punctuation z-section z-class z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">base_uri</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">https://google.com<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">middleware</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">request_logger</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">structlog</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">getLogger</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">response_as</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">html_response</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-function z-python"> <span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">search</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">self</span><span class="z-punctuation z-separator z-parameters z-python">,</span> <span class="z-variable z-parameter z-python">query</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-language z-python">self</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">request</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">get</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">/<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-variable z-parameter z-python">params</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">query</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p>Now the Google client takes a <code>Query</code> object. We can create parameter hierarchies, taking out a common <code>page</code> and <code>size</code> parameters and applying to all queries and enjoy code completion and type-correction.</p> <h2 id="summary">Summary</h2> <p>Formation is a generic functional middleware infrastructure for Python. With it, you compose a stack of middleware, each of which has a tiny API, a shared context, and an ability to cancel or proceed to the next middleware.</p> <p>In many ways, it is similar to Ruby’s Rack middleware, and Node’s connect — just for anything and not just the server side. In the context of Python, it is a higher-level abstraction over WSGI. Formation is not Pythonic, and it doesn’t abide to the Zen of Python; where suitable it does away with these, and optimizes for developer happiness.</p> <p>Feel free to <a href="https://jondot.github.io/formation/docs/docs">look through Formation’s docs</a> and have a go!</p> My Key Learnings after 30,000 LOC in Rust Mon, 29 Apr 2019 00:00:00 +0000 Unknown https://rng0.io/blog/my-key-learnings-after-30000-loc-in-rust/ https://rng0.io/blog/my-key-learnings-after-30000-loc-in-rust/ <p><img src="https://rng0.io/blog/my-key-learnings-after-30000-loc-in-rust/1.png" alt="" /></p> <blockquote> <p><strong>Like this article? Have questions?</strong> <a href="https://twitter.com/jondot"><strong>Follow me on twitter</strong></a> <strong>and ask away :)</strong></p> </blockquote> <p>I used to love C and C++. If we date back to the mid 90’s, I did C, probably poor C++ which I thought was great, and Assembly exclusively as part of my reverse engineering/security work.</p> <p>There’s something about being mindful of the price of instructions and data you use that feels intuitive and safe and generates confidence. You get low level building blocks, build your own blocks— and pay little for those because there’s a notion of “zero cost” or at least you directly control the cost. You had little moving parts and that — as we all know — is a strong trait of simple design.</p> <p>I also spent a good part of my career doing .NET (C#, F#) and the JVM (Java, Scala and much later, Kotlin as a shallower Scala). These are statically typed, garbage collected languages with what evolved to be state of the art virtual machines. I also did Ruby and modern versions of Javascript (Typescript, ESnext), or more generally programming languages which exhibit great developer experience; not surprisingly — it also led me to functional languages as well.</p> <p>Then, around 2010–2011, I found an outlier — Go. Not functional, not dynamic, and feels like home (home being C). When Node.js was failing me, I built and took Go based systems to production from early Go versions with extreme success (I remember that around a year later <a href="https://segment.com">Segment.io</a> publicized their move from Node.js to Go, as well as others).</p> <p>I think I felt that Go strikes the balance of robust systems programming and great performance with almost zero investment, and great developer experience. Some of that old 90’s feel, and a lot of that modern fantastic development practices.</p> <p>But there’s a reason I’m being nostalgic and making you read all that, which leads me to the first two things you should know about Rust.</p> <h3 id="1-you-have-to-be-ready-to-commit">1. You Have to be Ready To Commit</h3> <p>For me, Rust takes a stroll over the memory lane above and picks and drives home the best experiences from all those languages — which is a mind boggling, insane achievement.</p> <p>To pay for all that greatness(as I like to think), it comes with what people commonly experience as an annoyance — <a href="http://squidarth.com/rc/rust/2018/05/31/rust-borrowing-and-ownership.html">the borrow checker</a> and what is making a come back (looking at you gradual typing!) — static typing. This means that if you’re working in an environment with a GC at your disposal and / or a dynamic language, be ready to commit a good chunk of your time for thinking and designing around managing memory and types and for that experience in general — <a href="https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555">slow, safe thinking</a>.</p> <p>And here’s another curve-ball — it takes around 30 seconds to compile one of my larger Rust projects, and a type check cycle takes around 3 seconds. It’s so bad that as I switch from Rust to Go it takes my brain a few minutes to adjust to Go’s compliation speed; I type Go code and then gaze at the screen waiting for the compiler to catch up, only that it had caught immediately as I typed :).</p> <p>I’m sure you’ll find your own challenges; which brings me to the best Rust advice I can give you.</p> <p>Be ready to commit. Don’t expect to be in, write code, and win — especially not when you’re starting. Expect to be in, fall on your face, lots of face palms, write code, and be out after twice the amount of time you planned to invest (more like ten times but I didn’t want to sound gloomy).</p> <p>Which leads me to the second best Rust advice: survive the beginning.</p> <p>Or in the words of the great John Carmack, <em>Rust code feels very wholesome</em>.</p> <h2 id="2-rust-is-wholesome">2. Rust is Wholesome</h2> <p>If you’re planning to dive into Rust, here’s what you should expect starting up — if you’re coming from dynamic languages, static and garbage collected languages or static and manual memory managed languages.</p> <p>One of the major discussion points around how Rust “feels” is the borrow checker and ownership. Here’s my makeshift lifecycle of a Rust adopter and the borrow checker.</p> <ol> <li>Fight it</li> <li><em>Almost</em> lose it</li> <li>Agree with it</li> <li>Embrace it</li> <li>Understand it</li> </ol> <h3 id="coming-from-dynamic-languages">Coming From Dynamic Languages</h3> <p><img src="https://rng0.io/blog/my-key-learnings-after-30000-loc-in-rust/2.png" alt="" /></p> <p>This is a meaningful a change in how you design and build software. Because of the borrow checker and static typing, your design needs to be bolted into place firmly and there is little head room for “leaving things for later”.</p> <p>It means that from time to time, the borrow checker will tell you that your design stinks. Some times it doesn’t matter where you’ll shift code to satisfy the borrow checker — after hours of fighting and only after you take a few steps back and come to it later — you’ll realise your design was fundamentally flawed. You’ll negotiate your way around redesigns by cloning, copying, and <a href="https://doc.rust-lang.org/std/boxed/struct.Box.html">boxing</a>, and if you’re lucky it’ll be enough.</p> <p>If you’re more lucky you’ll actually go through redesign, which will force you to look at your object graph, responsibilities and dependencies, at which point you’ll finally <em>get</em> Rust.</p> <h3 id="coming-from-static-languages-garbage-collected">Coming From Static Languages, Garbage Collected</h3> <p><img src="https://rng0.io/blog/my-key-learnings-after-30000-loc-in-rust/3.jpg" alt="" /></p> <p>You will feel most of your pains with the borrow checker, the same as if you came from dynamic languages, only now you’ll have to unlearn parts of your static types intuition, and unlearning things is hard.</p> <p>Rust will feel bizarre and boilerplatey. And if you can’t get over it, at this point, try to challenge what you’re building — does it <em>have</em> to be built with Rust? Is that something you want to build with a systems programming language? Do you need a predictable memory model, zero cost, great performance (which arguably you can already get with a good VM, trading off memory for performance)? Maybe not. Better pick the right tool for the job.</p> <h3 id="coming-from-static-languages-manual-memory-management">Coming from Static Languages, Manual Memory Management</h3> <p><img src="https://rng0.io/blog/my-key-learnings-after-30000-loc-in-rust/4.png" alt="" /></p> <p>If you’re coming from C++, soon after getting into Rust you’ll feel like you’ve been handed a new pair of shoes; I say “soon after” and not “immediately after” because chances are your first impression would be that Rust “decides for you” too much, and then I would hope <a href="https://github.com/rust-lang/cargo">cargo</a> and time will do their thing and soften you up.</p> <p>Compared to C++, where you have (and want!) full control, some of Rust will feels like magic you won’t agree with. Also remember that Rust is <em>very</em> young, relatively; a good part of the ecosystem you’d expect to be there, coming from C++, doesn’t exist.</p> <h2 id="3-sweet-spot-for-building-tools">3. Sweet Spot for Building Tools</h2> <p><img src="https://rng0.io/blog/my-key-learnings-after-30000-loc-in-rust/5.png" alt="" /></p> <p>Building tools with Rust is amazing. You have an amazing <a href="https://clap.rs/">CLI library</a>, just as amazing <a href="https://serde.rs">serialisation library</a> for configuration, data structures, and a great <a href="https://docs.rs/log">logging library</a>, and even safe <a href="https://github.com/rayon-rs/rayon">parallelism for free</a>. <em>Safe</em> parallelism for free!</p> <p>The rest is battery included as the standard library is extremely rich.</p> <p>Rust is also performant and safe by default, has a very good cross-compilation story, and it wouldn’t hurt to have relatively compact binaries ranging 5mb-20mb (in most cases smaller than Go binaries) with jaw dropping start up times.</p> <h2 id="4-great-for-interop-and-replacing-existing-infra">4. Great for Interop and Replacing Existing Infra</h2> <p><img src="https://rng0.io/blog/my-key-learnings-after-30000-loc-in-rust/6.png" alt="" /></p> <p>In my mind, Rust’s killer feature was supposed to be C++ interop, but that’s only me. And like many languages that try to integrate with C++ and struggle, well, the short story is that <a href="https://users.rust-lang.org/t/any-updates-on-calling-c-from-rust/10257/2">it’s not fully there yet</a> (because of C++’s mangling etc.).</p> <p>However Rust’s FFI and integration direct-to-dynamic languages have <a href="https://neon-bindings.com/">become</a> <a href="https://usehelix.com/">unmatched</a> <a href="https://github.com/PyO3/pyo3">quickly</a>, which puts it well underway to replacing C/C++ as a low-level language for performance applications for dynamic languages — but that’s my own prediction.</p> <h2 id="5-it-s-consistent-and-stable">5. It’s Consistent and Stable</h2> <p>I started using Rust very early on, and to share a confession — ditched it with anger. I couldn’t get a build to work consistently with the language and crates and standard library breakage.</p> <p>When I picked it up again a couple years later, it wasn’t the same. I’ve never had a problem with crates or dependencies or packages, or build infrastructure! <a href="https://stephencoakley.com/2019/04/24/how-rust-solved-dependency-hell">and there is a good reason</a>.</p> <p>It takes that kind of polarized experience to appreciate stability as a highly valued feature.</p> <p>In a project with a large surface area in terms of crates, I’ve been consistently upgrading my nightly rust every weekend for the last half a year <em>without a single breakage</em>, and in fact — with gains in performance and compilation times!</p> <h2 id="6-slicing-and-dicing-software">6. Slicing and Dicing Software</h2> <p>One Rust feature that’s easy to overlook <a href="https://www.reddit.com/r/rust/comments/7o7ov7/can_someone_explain_how_features_work_in_cargo/">is features</a>. Features let you mix and match your software in <em>compile time</em>; you can ship reduced versions, differently licensed versions (using different crates with difference license), or ship with a menu for others to choose from, based on their own individual constraints.</p> <p>I use features as part of a plan to ship a commercial as well as a free version of software from the same codebase, which guarantees the free version binary has none of the licensed code, at all.</p> <p>I also like to control compilation times by excluding features I don’t use from other crates (more on that later).</p> <h2 id="7-amazing-performance">7. Amazing Performance</h2> <p>Clean, simple Rust is very fast by default:</p> <p>I see <a href="https://github.com/rust-lang/rust/pull/58623">performance gains</a> every time I update my Rust, so much that I’m looking forward and update Rust weekly (I use nightly).</p> <h2 id="8-use-go-for-when-the-ecosystem-isn-t-there">8. Use Go For When the Ecosystem Isn’t There</h2> <p><img src="https://rng0.io/blog/my-key-learnings-after-30000-loc-in-rust/7.png" alt="" /></p> <p>Some of the ecosystem isn’t there yet — and that’s fine. Obviously a better karma is to build and share what’s missing, but when pressed for time, I use Go to complete the Rust story, by <a href="https://github.com/vladimirvivien/go-cshared-examples">compiling go as</a> <code>cshared</code> and using <a href="https://docs.rs/libloading">libloading</a>. Go has a richer ecosystem, and I can iterate faster that way, when the trade-offs allow for it; you have to remember to account for build complexity (cross-compiling the Go code and making sure enough of it is statically linked to be loaded easily).</p> <p>Why Go of all others?</p> <ul> <li>Can be compiled into a clib, and has a fantastic cross compilation story around it.</li> <li>By now has an enormous ecosystem.</li> <li>Easy to pick up in a day, small number of concepts to learn and so a low mental overhead. Fast in, fast out, back to your Rust code.</li> <li>Shares some performance traits with Rust.</li> </ul> <h2 id="9-slow-compilation-speeds">9. Slow Compilation Speeds</h2> <p>In my mind this is a big one. And so I also want to share some tips to fix this.</p> <p>Starting with a <code>cargo new --bin</code> and jumping into your editor, compilation speeds are great and editor feedback is fast. Add around five crates, and you're in the 5 seconds zone for compilation.</p> <p>Fast forward to a medium to large project and you’re looking at 30 seconds for a build and around 5 seconds for a <code>cargo check</code>, which is what my editor (and probably other editors) will use for quick error checking, which is supposed to be what sets the squigglies for your fast feedback loop.</p> <h3 id="crate-shattering">Crate Shattering</h3> <p>To get a faster build you can break your project to pieces and compile each separately. I waited on this for a while, preferring to suffer with compilation times, because I didn’t believe it will go smoothly (but it did, massively smooth); I thought I’d be stepping into a Rust flavoured rabbit hole, somehow having the borrow checker have the upper hand on me in all this.</p> <p>I went from a painful 30 seconds to 9 seconds for a build by shattering the main project crate into four (read: split the main code base to multiple independent libraries), most of that 9 seconds now is linking a binary. And it turns out, it was a breeze.</p> <p>Additional benefits I’ve discovered with the new project layout:</p> <ul> <li>Easy to identify optional features.</li> <li>Easy to flag some of the sub crates as features and not include them.</li> <li>Easier to isolate and performance test smaller crates.</li> </ul> <p>So, lots of power when breaking to small pieces. This means project layout becomes much more important in Rust, as it is a workflow optimization enabler for individuals and more so for teams.</p> <p>To break down cleanly into crates, as with other languages and <a href="https://blog.ndepend.com/hexagonal-architecture/">modern software architecture</a>, try to identify your:</p> <ul> <li>Core: data, types, abstractions</li> <li>Logic: algorithms (can sit with core)</li> <li>Peripherals: networking, HTTP, reporting</li> <li>App: entry point, uses and glues everything</li> </ul> <p>And then follow projects like <a href="https://github.com/BurntSushi/ripgrep">ripgrep</a> to actually see how to break down your project using cargo <a href="https://doc.rust-lang.org/book/ch14-03-cargo-workspaces.html">workspaces</a>.</p> <p>Here’s another nice surprise that happens which reminds you Rust is smart:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">4 | use crate::project::Project; </span><span class="z-text z-plain"> | ^^^^^^^ </span><span class="z-text z-plain"> | | </span><span class="z-text z-plain"> | unresolved import </span><span class="z-text z-plain"> | help: a similar path exists: `aural_engine::project` </span></code></pre> <p>Rust will help you if it sees import paths in your code that looks for local code but now that you’ve moved those into a crate — will point to it.</p> <p>Note that to enjoy speed, you might have to open your editor <em>rooted at one of the sub crates</em>. This means it’ll do checks based on that crate; if you open at workspace root, your editor (or cargo) will do checks in the context of all dependencies. Namely, if B depends on A, and B changes — cargo/rust will check both B and A, nullifying the purpose of crate shattering. However if you open your project in the context of B, and change B, only B will be checked, which means — speed.</p> <p>To finish with some reading material about diagnosing, thinking and fixing performance, here’s <a href="https://vfoley.xyz/rust-compile-speed-tips/">some more compiler tips</a>, <a href="https://github.com/rust-lang/rust/issues/48547">issue tracking progress</a>, and how to <a href="https://blog.mozilla.org/nnethercote/2016/10/14/how-to-speed-up-the-rust-compiler/">profile the compiler</a> and some more <a href="https://www.worthe-it.co.za/programming/2018/09/23/performance-tuning-in-rust.html">profiling and benchmarking</a>.</p> <p>For better linking times, experiment with <a href="https://lld.llvm.org/#performance">linking with LLD</a> on your OS. Sadly, many of these aren’t possible on macOS (including linking with LLD). To give you a clue about how to profile on OSX (since it does not have <code>perf</code>) <a href="http://carol-nichols.com/2017/04/20/rust-profiling-with-dtrace-on-osx/">read this</a> or more simply use <a href="https://github.com/tcr/macos-profiler">macos-profiler</a> that packages these instructions more or less.</p> <h2 id="10-services-story-is-still-in-progress">10. Services Story is Still In Progress</h2> <p>Servicing, APIs and web frameworks can be a dream in Rust, but the infrastructure story to support it is still being told.</p> <p>A few crates, which I find exciting are <a href="https://github.com/hyperium/hyper">hyper</a>, <a href="https://github.com/tokio-rs/tokio">tokio</a> and <a href="https://github.com/actix/actix-web">actix</a>. I’ve settled on actix for services, but still, I recognize that this is not what will make me pick Rust for a traditional backend service project at this stage.</p> <h2 id="bonus-documentation-is-great">Bonus: Documentation Is Great</h2> <p>There’s a high chance you already know this, which is why I didn’t count it. The consensus is that Rust has one of the better documentation stories, and again to factor in the number of years that Rust exists — it’s nothing short of amazing. You have <a href="https://doc.rust-lang.org/edition-guide/index.html">Rust editions</a>, <a href="https://doc.rust-lang.org/rust-by-example/index.html">by example</a>, <a href="https://doc.rust-lang.org/1.30.0/book/first-edition/index.html">effective Rust</a> (although in the older book), and the regular documentation you’d find though googling anything Rust related, AKA <a href="https://doc.rust-lang.org/book">“the book”</a>.</p> <p>If I ignore the fact that Rust is still new, I’d be happy to see a <a href="http://learnyouahaskell.com/">learnyou</a> variant in Rust.When you’re ready, check out my other Rust related article (part of this series), <a href="https://rng0.io/blog/12-killer-rust-libraries-you-should-try/">that covers great Rust libraries</a> that you should try.If you like developer tools, productivity and security — we’re looking for a great chat, and/or great people to join us! drop us a line: <a href="mailto:hello@spectralops.io">hello@spectralops.io</a></p> 12 Killer Rust Libraries You Should Try Fri, 19 Apr 2019 00:00:00 +0000 Unknown https://rng0.io/blog/12-killer-rust-libraries-you-should-try/ https://rng0.io/blog/12-killer-rust-libraries-you-should-try/ <p>I’ve just crossed a 20K LOC in one of my bigger Rust projects, and thought about pausing for a moment and sharing some great Rust libraries that I’ve used.</p> <p>I also plan to cover topics such as deciding between Rust vs Go, or deciding if to adopt Rust for your team or project, and what should you expect after you’ve decided to take it on, in a future article.</p> <p>UPDATE: the second part is now out: <a href="https://rng0.io/blog/my-key-learnings-after-30000-loc-in-rust/">10 key learnings in Rust after 30,000 lines of code</a></p> <h2 id="clap">clap</h2> <p>Building CLI tools in Rust is a match made in heaven — check out <a href="https://github.com/BurntSushi/ripgrep">ripgrep</a> and Rust’s own <a href="https://crates.io/">Cargo</a>. Hyperfast start up time, small binary size, type safe code, runtime safe binary, <a href="https://github.com/japaric/rust-cross">cross compiling</a> to almost every architecture you might want.</p> <p>To start building CLI tools, <a href="https://clap.rs/">clap</a> is a fantastic CLI library, it’s so good I don’t see any reason to have a dozen alternatives like common in other languages. In fact, if you want to try Rust, I would recommend taking a look at <a href="https://rustup.rs/">rustup</a> and then trying out this library and see where it gets you.</p> <p><img src="https://rng0.io/blog/12-killer-rust-libraries-you-should-try/1.png" alt="" /></p> <h2 id="serde">serde</h2> <p>Like clap, <a href="https://serde.rs">serde</a> is a feature rich and performant generic serialization library. In fact, thinking about Java and .NET, I don’t remember a serialization library that was this well made from all aspects — ergonomics <em>and</em> performance.</p> <p>Don’t try reading/writing from/to files on your own, instead — write your data types first and make serde do all the work. As a bonus you can mix and match the data format (YAML, JSON) after everything is done.</p> <p><img src="https://rng0.io/blog/12-killer-rust-libraries-you-should-try/2.png" alt="" /></p> <h2 id="reqwest">reqwest</h2> <p><a href="https://docs.rs/reqwest/0.9.15/reqwest/">Reqwest</a> follows the gold standard for HTTP client libraries like <a href="https://github.com/request/request">request</a>, <a href="https://github.com/visionmedia/superagent">superagent</a> and <a href="http://docs.python-requests.org">requests</a> and applies it to Rust perfectly. It’s my go-to library for HTTP clients, is feature packed and complete.</p> <p><img src="https://rng0.io/blog/12-killer-rust-libraries-you-should-try/3.png" alt="" /></p> <h2 id="rayon">rayon</h2> <p><a href="https://github.com/rayon-rs/rayon">Rayon</a> is a “data parallelism library for Rust” or in simple words, give it data and it will know how to split it into independent chunks and work all your CPU cores.</p> <p>Or even more simply, give it a list and it’ll parallelize <code>map</code> over it, amonst other features. Extremely useful for CLI tools; not all languages are good with parallelism over the command line.</p> <p><img src="https://rng0.io/blog/12-killer-rust-libraries-you-should-try/4.png" alt="" /></p> <h2 id="slog-and-log">slog and log</h2> <p><a href="https://github.com/slog-rs/slog">slog</a> is a very complete logging <em>suite</em> for Rust. It is a core followed by many plugins such as <a href="https://docs.rs/slog-term/">term</a> for terminal output, <a href="https://docs.rs/slog-json/">json</a> for JSON output and more.</p> <p>You should know that there is also <a href="https://docs.rs/log">log</a> which aims to be part of standard Rust, and is a simpler alternative. I’ve personally switched to log from slog for this reason.</p> <p><img src="https://rng0.io/blog/12-killer-rust-libraries-you-should-try/5.jpg" alt="" /></p> <h2 id="itertools">itertools</h2> <p>Wouldn’t hurt to have a few more operators over your lists would it? especially as many or most of these are zero cost. With <a href="https://github.com/bluss/rust-itertools">itertools</a> you get that. Great if you’re a fan of libraries like <a href="https://lodash.com">lodash</a>.</p> <p><img src="https://rng0.io/blog/12-killer-rust-libraries-you-should-try/6.png" alt="" /></p> <h2 id="hyper">hyper</h2> <p><a href="https://hyper.rs/">hyper</a> is a fast HTTP implementation written in and for Rust (as opposed to those written in C, that cover for performance in dynamic languages). You’ll find hyper to be in almost every high level library that you use, and if you use it directly, it feels a bit like <a href="https://netty.io/">Netty</a> or <a href="https://twitter.github.io/finagle/">Finagle</a>. I found myself treating hyper both as an HTTP toolbox (using pieces from it) and as a whole, building a server over it.</p> <p><img src="https://rng0.io/blog/12-killer-rust-libraries-you-should-try/7.png" alt="" /></p> <h2 id="actix">Actix</h2> <p>Guess what <em>doesn’t</em> use hyper? <a href="https://github.com/actix/actix-web">Actix</a>. Actix tries to be simpler, and from my experience — it delivers. Often I use Actix instead of Hyper because it’s more highlevel and for service purposes — more mature. Today, I default to using Actix directly over Hyper, unless I need to build something low level, or have a library that requires Hyper directly (there’s many).</p> <p><img src="https://rng0.io/blog/12-killer-rust-libraries-you-should-try/8.png" alt="" /></p> <h2 id="pyo3">PyO3</h2> <p><a href="https://github.com/PyO3/pyo3">PyO3</a> is one of the more popular libraries for building Rust libraries in Python (or is it Python libraries in Rust? 😀). If you want to see a result of such a mix, here’s <a href="https://github.com/mre/hyperjson">hyperjson</a> — a JSON library for Python backed by Rust’s serde.</p> <p>By relying on Rust’s safety and serde’s performance, you get a safe <em>and</em> one of the fastest JSON libraries for Python in almost no effort.</p> <p>Want to improve Python in 3 steps?</p> <ol> <li>Find a great Rust library</li> <li>Wrap it with PyO3</li> <li>Profit!</li> </ol> <ul> <li>Note: PyO3 (and possibly rust-cpython) may have some design flaws that may forgo safety. <a href="https://www.reddit.com/r/rust/comments/azit15/i_made_a_super_simple_example_guide_of_how_to/ei89s8e/?context=3">Read here</a> for more details.</li> </ul> <p><img src="https://rng0.io/blog/12-killer-rust-libraries-you-should-try/9.png" alt="" /></p> <h2 id="proptest">proptest</h2> <p><a href="https://lib.rs/crates/proptest">proptest</a> is a property based testing library for Rust. Ever since I’ve used <a href="http://hackage.haskell.org/package/QuickCheck">QuickCheck</a> in my short Haskell stint, I’ve looked for these kind of libraries in every language I’ve used — libraries that propose that they’ll find a failing test case for you automatically by intelligently thinking up a set of input data that trips up your code.</p> <p><img src="https://rng0.io/blog/12-killer-rust-libraries-you-should-try/10.png" alt="" /></p> <h2 id="libloading">libloading</h2> <p>For those of you who would like to mix Go or any other c-lib library into their Rust frontend, <a href="https://docs.rs/libloading/0.5.0/libloading/">libloading</a> make this simple.</p> <p>Building medium to large projects with Rust the last year, I accept that some parts of the Rust ecosystem aren’t ready yet, and am not shy of building these in other languages (mostly Go)— only to hook everything back into Rust with libloading.</p> <p><img src="https://rng0.io/blog/12-killer-rust-libraries-you-should-try/11.png" alt="" /></p> <h2 id="performance">Performance</h2> <p>Last thing you should know is that clean, simple Rust is very fast by default. Here’s a talk about that at the correct time:</p> <p>There are a couple more libraries I like to use almost always, which give you a nice performance boost out of the default that Rust already providers:</p> <ul> <li><a href="https://docs.rs/regex/1.1.6/regex/">regex</a> — a Regex library that’s <em>really</em> fast, including compared to other programming languages.</li> <li><a href="https://crates.io/crates/jemalloc-sys">jemalloc</a> — one of the fastest memory allocators, that was dropped in recent Rust versions in favor of operating system default. I found that it’s faster than the default OSX allocator, but use it only if you know that you need it since it adds some weight to your binaries (around 700kb).</li> </ul> <h3 id="summary">Summary</h3> <p>Rust, in one word is — Power.</p> <p>But there’s much more to say. For example — Rust is also extremely young compared to other programming languages.</p> <p>Read more about <a href="https://rng0.io/blog/my-key-learnings-after-30000-loc-in-rust/">my experience with Rust</a>.</p> 4 Ways to Test Your APIs End to End Thu, 28 Mar 2019 00:00:00 +0000 Unknown https://rng0.io/blog/4-ways-to-test-your-apis-end-to-end/ https://rng0.io/blog/4-ways-to-test-your-apis-end-to-end/ <p><img src="https://rng0.io/blog/4-ways-to-test-your-apis-end-to-end/1.png" alt="" /></p> <p>Here’s a series of steps for such test flow for your common note-taking app:</p> <ol> <li>Log in</li> <li>Save a note</li> <li>Get a note</li> <li>Delete a note</li> </ol> <p>A “Get” action can counting on data to exist. A step might expect you to authenticate such as the “Save a note” step, therefor having to perform an authentication step that supplies, for example, a <a href="https://jwt.io/">JWT token</a>.</p> <p>And, trivially, to log in — someone should have put a user with your credentials in the database before all this was able to run.</p> <p>You can test this flow using white-box — your integration tests, with access to system source, being able to mock things out and build helpers to promote testing, or black-box — no access to your source, having just an opaque service address.</p> <h2 id="if-everything-was-perfect">If Everything Was Perfect</h2> <p>In theory, you should not be doing black-box end-to-end multi request flows. That is, in <em>theory</em>, you should test each part in isolation and then test integration generically (mocking some parts of your system).</p> <p>However, when you don’t control the test subject, you <em>have</em> to exercising black-box testing which means — multi request flows.</p> <p>Multi request flows pose a challenge — you’re no longer testing a section of your product or API in isolation of its dependencies.</p> <p><strong>A Word About Exception Cases</strong></p> <p>In some cases, we can’t perform multi request flows without a price. For example, if you’re testing a Fintech or ECommerce product, and you want to run a flow that at some point charges a credit card, and you’re black-box testing, you have to have a credit card ready.</p> <p>Given enough of these tests running with a real card and real money (charging minimal amounts), when someone drops a money laundering question about your tests — which they will — it becomes borderline impossible to test this way.</p> <h2 id="the-current-state-of-api-testing-tools">The Current State of API Testing Tools</h2> <p>The discussion around end to end testing tools for APIs is as the one for doing <a href="https://www.seleniumhq.org/">Selenium</a> tests for Web apps. Both have the challenges of black-box testing: flaky tests, test-suite prepwork, setup and teardown, side-effects, inconsistencies and, lastly, tooling.</p> <p>To understand the current state of tooling, the first step is to come up with a good evaluation framework:</p> <ul> <li><strong>Self-contained</strong> — can we run this tool as a standalone tool, regardless of our programming language or of the test subject tech stack? With the idea of using one definition with one kind of tool through product lifecycle, from development to production.</li> <li><strong>Integration tests</strong> — can we use it to build plain integration tests? Ideally, to promote reuse, we should use the same tool for white-box integration tests while doing high-level black-box multi request flows.</li> <li><strong>Test definition as configuration</strong> — can someone that is not skilled in the test subject tech stack add new tests? is it easy? is it quick?</li> <li><strong>Variable passing</strong> — multi request flows is about sharing context. This means a test framework supports passing step results from one onto the other.</li> <li><strong>Set up and tear down</strong> — does it support doing database set up before running the tests? tear down? is it generic in the sense that we can run anything before a test?</li> <li><strong>Contract validation</strong> — does it validate the contract against a schema?</li> <li><strong>Content validation</strong> — does it validate content?</li> </ul> <h3 id="supertest">Supertest</h3> <p>Starting with <a href="https://github.com/visionmedia/supertest">Supertest</a> — one of the most popular integration testing library for Javascript and Node.js. You use it on top of your existing testing framework, and it offers a nice fluent-interface for defining API tests (black-box or white-box).</p> <p><img src="https://rng0.io/blog/4-ways-to-test-your-apis-end-to-end/2.png" alt="" /></p> <p>❎ <strong>Self-contained</strong>— No. Has to have a testing framework to drive it.</p> <p>✅ <strong>Integration tests</strong> — Yes.</p> <p>✳️ <strong>Test definition as configuration</strong> — No. But if you really squint you <em>can</em> look at its fluent-interface as a DSL for a configuration language.</p> <p>✅ <strong>Variable passing</strong>— Yes. Naturally, you’re running it in Javascript.</p> <p>✅ <strong>Set up and tear down</strong> — Yes. Part of your testing framework.</p> <p>❎ <strong>Contract validation</strong> — No.</p> <p>✳️ <strong>Content validation</strong> — Yes. But you have to explicitly verify it in your tests.</p> <p>Notable mention: <a href="https://github.com/vesln/hippie">hippie</a> which has a similar approach.</p> <h2 id="dredd">Dredd</h2> <p><img src="https://rng0.io/blog/4-ways-to-test-your-apis-end-to-end/3.png" alt="" /></p> <p><a href="https://dredd.readthedocs.io/en/latest/">Dredd</a> is a tool backed by <a href="https://apiary.io/">Apiary</a> (now part of Oracle). It is <em>“a language-agnostic command-line tool for validating API description document against backend implementation of the API.”</em>.</p> <p>It can take <a href="https://apiblueprint.org/">API Blueprint</a> or <a href="https://swagger.io">Swagger</a> (now OpenAPI) descriptions for your API and validate (aka test) your live services or service under test.</p> <p>✅ <strong>Self-contained</strong>— Yes.</p> <p>✅ <strong>Integration tests</strong> — Yes.</p> <p>✅ <strong>Test definition as configuration</strong> — Yes. Can take both Swagger and API Blueprint format, which are widely popular.</p> <p>❎️ <strong>Variable passing</strong> — Partial but listed as “No” because it tends to become a mess very quickly. Solution feels nontrivial/fragile (coordinating results via hooks file).</p> <p>✅ <strong>Set up and tear down</strong> — Yes. Via hooks feature.</p> <p>✅ <strong>Contract validation</strong> — Yes.</p> <p>✳️ <strong>Content validation</strong> — Partial. Depending on whether you use Swagger or API Blueprint, and their levels of support for “contract examples”, which is not perfect.</p> <h2 id="newman-and-postman">Newman (and Postman)</h2> <p><img src="https://rng0.io/blog/4-ways-to-test-your-apis-end-to-end/4.png" alt="" /></p> <p><a href="https://github.com/postmanlabs/newman">Newman</a> is <em>“The command-line companion for Postman”</em>. It means that if you use <a href="https://www.getpostman.com/">Postman</a> as an individual or a team, you can now reuse your Postman collections as API integration and end-to-end flow tests.</p> <p>It can take its own proprietary format or <a href="https://swagger.io">Swagger</a> (now OpenAPI) descriptions for your API tests and validate (aka test) your live services or service under test, using scripts to do the final validation. This means you’re maintaining a way to describe API calls and a way to validate expectations.</p> <p>✅ <strong>Self-contained</strong> — Yes.</p> <p>✅ <strong>Integration tests</strong>— Yes.</p> <p>✳️ <strong>Test definition as configuration</strong>— Partial. You have to maintain the validation part by writing scripts, which is kind of awkward — why not flip this around and write tests in code?</p> <p>✅ <strong>Variable passing</strong>— Yes. But still a bit of a cranky solution because you have to maintain code in Postman UI that does this.</p> <p>✅ <strong>Set up and tear down</strong>— Yes. When using Newman as embedded.</p> <p>❎ <strong>Contract validation</strong> — No.</p> <p>✅️ <strong>Content validation</strong>— Yes. Again pending you do that in your tests.</p> <h2 id="pact">Pact</h2> <p><img src="https://rng0.io/blog/4-ways-to-test-your-apis-end-to-end/5.jpg" alt="" /></p> <p>Pact is a different kind of tool. It is a popular implementation of <a href="https://martinfowler.com/articles/consumerDrivenContracts.html">Consumer Driven Contracts</a> which makes it an API testing tool.. kind of.</p> <p>It feels that Pact is interested in providing tooling for verifying the consumer of the API more than in the API itself, how ever, there’s a way to test both sides — see that the consumer is behaving against a Pact, and that the provider is as well, which makes it an API testing tool, kind of.</p> <p>❎ <strong>Self-contained</strong> — No.</p> <p>✳️ <strong>Integration tests</strong> — Kind of. It depends how you define “integration”, either case it’s not how we define it here.</p> <p>✳️ <strong>Test definition as configuration</strong>— Kind of. It looks like configuration but it isn’t really; Originally was a Ruby based DSL, and now there’s support for Javascript, Go, JVM and more.</p> <p>✅ <strong>Variable passing</strong> — Yes.</p> <p>✅️ <strong>Set up and tear down</strong> — Yes.</p> <p>❎ <strong>Contract validation</strong>— No.</p> <p>✅️ <strong>Content validation</strong>— Yes.</p> <h2 id="honorable-mention-record-replay-testing">Honorable mention: Record/Replay Testing</h2> <p><img src="https://rng0.io/blog/4-ways-to-test-your-apis-end-to-end/6.png" alt="" /></p> <p>Nicknamed “VCR testing” and made hugely popular by the Ruby <a href="https://github.com/vcr/vcr">VCR gem</a> this technique requires to first record a set of API interactions, and then verify the consumer code (similar to Packt) against a recorded version of the API.</p> <p>As time moved, the idea spread and today we have such libraries for Python, Go, Node.js, Java, and some read the same original “Cassette” recording format the original Ruby gem produced.</p> <p>In this way, again, we’re not really testing integration in the way we define it to be here. However, a long while ago I built <a href="https://github.com/jondot/rcv">RCV</a> which reverses VCR and tests the recordings <em>against the provider</em>, which makes it completely compatible with Pact’s feature-set.</p> <h3 id="conclusion">Conclusion</h3> <p>There is still space for a testing tool and framework that can hit all cylinders. One that has tests spec’d out as configuration, can be stand-alone, can cover development as well as pre-production and canary testing, and have the black-box testing quirks such as test flow context, set up and tear down all easy to do.</p> <p>As a general practice, if I had to choose from the list above, focusing on test flow format, I would always take a tool that can read a Swagger / OpenAPI format, or if such a tool has its own format — it has to be worth it, and it has to be a simple enough format to allow to build tooling on top of.</p> How to Test a Full React App Using Nothing But Storybook Wed, 27 Feb 2019 00:00:00 +0000 Unknown https://rng0.io/blog/how-to-test-a-full-react-app-using-nothing-but-storybook/ https://rng0.io/blog/how-to-test-a-full-react-app-using-nothing-but-storybook/ <p>In my <a href="https://rng0.io/blog/how-to-snapshot-test-everything-in-your-redux-app-with-jest/">previous article about how to test everything in your Redux app with Jest</a> I’ve discussed how you can use snapshot testing to test Redux actions, reducers, views and side effects.</p> <p>We’ll divide this subject into a two-part series:</p> <ol> <li><strong>Part 1 (you’re reading it!):</strong> Testing theory, what changed in the world, and how to use storybook to cover your bases with Storyshots, Puppeteer, and Storybook-Cartesian</li> <li><strong>Part 2 (follow this publication to be notified when its out!):</strong> Applying the same practices to multi-browser testing with Storybook-Selenium.In this article we’ll focus on the React side of things, and showcase part of the testing approach at <a href="https://hiredscore.com">HiredScore</a>.</li> </ol> <p>Here’s a question: if you had the constraint of picking <em>just one</em> of the common test layers you usually use with every feature, which one would you choose that would give you the most ROI if it is the only one that exists (return on investment)?</p> <ol> <li><strong>Unit tests</strong>, with a good coverage</li> <li><strong>Integration tests</strong>, visual regression</li> <li><strong>End-to-end tests</strong>, cross-browser visual regression</li> </ol> <p>What if I told you that it’s possible to write <em>no test at all</em> and get (3), (2) and most of (1)?</p> <p>In fact, you’ll be writing a <a href="https://storybook.js.org">Storybook</a> story, not a test. And if you already are in the <em>very healthy</em> habit of writing Storybook stories for your development and design handoff workflow — then you’d be writing <em>no tests at all</em>.</p> <p>This means, you’ll have much more time on your hands to do proper TDD for domain logic!</p> <h3 id="the-testing-pyramid">The Testing Pyramid</h3> <p><img src="https://rng0.io/blog/how-to-test-a-full-react-app-using-nothing-but-storybook/1.png" alt="" /></p> <p>The testing pyramid was a concept that got a limelight during the gold-rush of testing; years 2006 to 2009 gave birth to BDD (Behavior Driven Development), cucumber, modern and humane automated testing, and sparked the culture of testing that was getting a healthy boost from the Ruby community, which kept looking at software as craft and quality as a first-class citizen.</p> <p>One example was Ruby on Rails and its massive opinionated push towards testing and test strategy and tooling <a href="https://twitter.com/dhh/status/472072157821169664">right out of the box</a>. Pretty much shaping everything we know about good testing today.</p> <p>At the base of the pyramid we have our unit tests; small, quick, isolated tests that can run en-masse. As we go up, the complexity and run time of tests goes up as well, and so integration tests and then end-to-end tests come next, and then manual tests or the so called “eyeball tests” come last.</p> <p>What’s important here is that as you go up — you should ideally write less tests of that certain level you went up to.</p> <p>With that metaphor comes its arch-enemy — the Ice Cream Cone:</p> <p><img src="https://rng0.io/blog/how-to-test-a-full-react-app-using-nothing-but-storybook/2.png" alt="" /></p> <p>In this model, you have a massive amount of manual tests and close to zero unit tests. The worst kind of situation to be in for agility and reliability.</p> <h3 id="introducing-the-testing-diamond">Introducing: The Testing Diamond</h3> <p><img src="https://rng0.io/blog/how-to-test-a-full-react-app-using-nothing-but-storybook/3.png" alt="" /></p> <p>Having React coupled with Storybook and Storyshots, unlocks a different model: the Diamond model.</p> <p>The diamond model <strong>for your UI/App</strong> means: little to zero unit tests, massive amount of integration tests, and zero manual tests.</p> <p>What changed? Integration tests were avoided in the early days because they had a reputation of running slowly; granted — with most technologies this is still very true.</p> <p>With Jest, React, and Storybook/Storyshots, this is (arguably) no longer the case. No longer must you bring up a browser for each test that leaves its traces in your test environment, or have flaky test suites run and fail randomly, using a not-so-smart test runner that forces you to run <em>everything</em> exactly when you didn’t want to. It’s an era where frontend tooling really does work, and hard becomes easy.</p> <h2 id="redefining-integration-tests">Redefining Integration Tests</h2> <p>Before we see how Storybook and friends checks all of our testing boxes, it makes sense to align our testing terminology in the context of React. A unit test may mean many different things to different people or different teams, and even more so integration tests may mean different things as well.</p> <p>For example, I view a unit tests as something that tests a module in isolation — and I don’t really care about if that module is a <em>class</em>, a <em>function</em> or a set of highly cohesive classes that sit together in a module. Others might argue differently, which is fine.</p> <p>I also view an integration test as anything that connects one or a number of such modules with the external world for a wide definition of “external world”. For example, rendering a single React component naively is a unit test in my book. But rendering that same component with various dependencies such as CSS, fonts, wrapper component, theme/styling support and so on (still just showing that single component) is an integration test — even if we don’t combine multiple components with one another, we <em>are</em> combining a single component with some infrastructure facilities — which are external to it.</p> <p>Let’s line up our solutions and keep track of this table as we go:</p> <p><img src="https://rng0.io/blog/how-to-test-a-full-react-app-using-nothing-but-storybook/4.png" alt="" /></p> <h3 id="storybook-driven-development">Storybook Driven Development</h3> <p><img src="https://rng0.io/blog/how-to-test-a-full-react-app-using-nothing-but-storybook/5.png" alt="" /></p> <p>In a modern React workflow, you have Storybook integrated and driving your work. Which means you build a component storybook-first, in a storybook-driven-development fashion.</p> <p>React components are largely a functional beast. They have inputs and outputs and no side effects. What this means is that for a process that takes an input and produces an output — the effective way to test would be to set up a <em>test harness</em>; something that takes a module (test subject), automatically provides input, runs the module and automatically verifies the output.</p> <p><a href="https://jestjs.io/docs/en/snapshot-testing">Jest snapshots</a> is such a test harness. Jest snapshots in the context of Storybook are what <a href="https://storybook.js.org/testing/structural-testing/"><em>Storyshots</em></a> <em>is.</em></p> <h2 id="storyshots-for-structural-tests">Storyshots for Structural Tests</h2> <p>If you already write stories for every component, you already are writing tests, and you just don’t know it yet. Given the thesis above, each of your stories can automatically become a tests:</p> <ul> <li>Input is your story</li> <li>Processing is simply rendering a story (which storybook already does)</li> <li>Output is a generated snapshot</li> </ul> <p>And this is what <a href="https://github.com/storybooks/storybook/tree/master/addons/storyshots/storyshots-core">Storyshots does</a>. Storyshots will verify that a React component renders correctly; and if you build multiple stories with a number of different properties then Storyshots can snapshot those as well, and those would be verified on every test run.</p> <p><img src="https://rng0.io/blog/how-to-test-a-full-react-app-using-nothing-but-storybook/6.png" alt="" /></p> <h2 id="storyshots-and-puppeteer-for-visual-tests">Storyshots and Puppeteer for Visual Tests</h2> <p>With Storyshots, we’ve covered the DOM. However, in some cases snapshots are not enough; these cases are often browser driven — an example would be an unexpected cascading (CSS) set of rules that step on each other, leading to a different rendering from what we expected.</p> <p>Rendering a story to pixels is beneficial, especially when it too comes for free. With <a href="https://github.com/storybooks/storybook/tree/master/addons/storyshots/storyshots-puppeteer">Storyshots Puppeteer</a> this easily becomes the case.</p> <p><img src="https://rng0.io/blog/how-to-test-a-full-react-app-using-nothing-but-storybook/7.png" alt="" /></p> <h2 id="storyshots-cartesian-for-10x-testing-impact">Storyshots Cartesian for 10x Testing Impact</h2> <p>Now that we have three layers of defense ready to snapshot and use stories from your storybook set up, why not give it more stories? For example, looking at this component:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">Button</span> <span class="z-variable z-other z-readwrite z-ts">disabled</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">false</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">highlighted</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">false</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">Click</span> <span class="z-variable z-other z-readwrite z-ts">Me</span> </span><span class="z-source z-ts"><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">Button</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></code></pre> <p>Reveals that we have a few creative ways to verify that this button is correct:</p> <ol> <li>A button that’s disabled should look like it’s disabled</li> <li>Same for when it’s highlighted</li> <li>A button that has an empty text, should be still visible and look like a button (i.e. will not squash down to be one pixel wide), there should be some minimal width for it</li> </ol> <p>And each of these can be a story. This works great because the moment we add a story, we have three snapshot tools that go off:</p> <ol> <li>“Regular” snapshots — take care of snapping the DOM</li> <li>Puppeteer snapshots — makes sure a browser renders correctly, visual regression</li> <li>Multibrowser snapshots — visual regression on old browsers, old IE, a combination of Mobile+Browser variants that your customers use frequently, and so on.</li> </ol> <p>But, what about a combination of the stories? what about a button that’s highlighted <em>and</em> disabled?</p> <p>For this, we employ <a href="https://github.com/jondot/storybook-cartesian">Storybook Cartesian</a>. With it, we can state the various single-property options for a prop, and it will generate all of the required combinations <em>as fully functional stories</em>; and so, these too will be immediately picked up by our multiple testing stages.</p> <p><img src="https://rng0.io/blog/how-to-test-a-full-react-app-using-nothing-but-storybook/8.png" alt="" /></p> <p>Which creates:</p> <p><img src="https://rng0.io/blog/how-to-test-a-full-react-app-using-nothing-but-storybook/9.gif" alt="" /></p> <p>And the resulting table gets a little bit extra for everything!</p> <p><img src="https://rng0.io/blog/how-to-test-a-full-react-app-using-nothing-but-storybook/10.png" alt="" /></p> <h2 id="interaction-tests-the-lone-survivor">Interaction Tests: The Lone Survivor</h2> <p>By using our existing stories and these techniques we’ve got a layers on top of layers of defense. What none of those cover is <em>interaction</em> tests and <em>stateful</em> tests, which you can now do sparingly and treat as an edge case and not the norm.</p> <h2 id="conclusion">Conclusion</h2> <p>What do we lose by doing a <em>little</em> amount of unit tests but a great amount of integration test?</p> <p>Well, on “paper” we lose the ability to pinpoint failing code very quickly. A unit test effectively splits our code to small units (when done right), and so when a test fails we should get a good intuition for what unit failed and what line of code was it that triggered the failure.</p> <p>We don’t get that with integration tests, but if we think about React components, it doesn’t matter. In any case these are mostly <em>rendering tests</em> so even if we had such unit tests we would still ask a question that relates to rendering of a component — “what CSS did we get wrong?” or “what markup should we fix?”. In that sense — integration tests and snapshot tests are as effective in fixing a failing test as a unit test.</p> <p>That said, <em>you should definitely</em> keep your “classic” unit tests for logic, library and domain model code. All these things you put in <code>/lib</code>, external packages that deal with your domain model and so on.</p> <p>Using <a href="https://flow.org/">Flow</a> or <a href="https://www.typescriptlang.org/">Typesript</a> is another good thing you can do when doing this kind of testing. Null safety, and type errors kind of test failures would not even get a chance to become a failing test and so increases the effectiveness of a healthy integration layer.</p> <p>As with anything, don’t apply these principles blindly to every situation — although it should work quite well in the context described (React components, rendering, views, browsers).</p> Shipping Rust Binaries with GoReleaser Sat, 16 Feb 2019 00:00:00 +0000 Unknown https://rng0.io/blog/shipping-rust-binaries-with-goreleaser/ https://rng0.io/blog/shipping-rust-binaries-with-goreleaser/ <p><img src="https://rng0.io/blog/shipping-rust-binaries-with-goreleaser/1.png" alt="" /></p> <p>As consumers, we like to get a binary release in one of the following ways:</p> <ul> <li>OS native package format such as <code>deb</code>, <code>rpm</code> and so on.</li> <li>Language native package format such as <code>pip</code>, <code>npm</code> and so on. It's not common to supply a pure binary artifact this way but some do it, such as <a href="https://www.npmjs.com/package/puppeteer">puppeteer</a> which downloads a latest binary of a headless Chrome.</li> <li>A curl shell combo such as <code>curl http://acme.org/bin | sh</code> which is fast, has no dependencies, but receives criticism because you're essentially letting your users execute remote code. This is why I avoid these, but if I must, verify the shell script by hand first.</li> <li>A modern binary packager such as <a href="https://brew.sh/">Homebrew</a>.</li> </ul> <p>While the process may differ from one technique to another, they all require a way to host the binaries or packages, except perhaps the language-native package formats, where you can get away with pushing large binaries into the registries (but it may be frowned upon).</p> <p>In principle, these are the steps required do package and release:</p> <ol> <li>Cross compile your binaries (e.g. Linux, Windows, macOS), since we’re distributing binaries we have to have a ready-made version per platform and architecture.</li> <li>Package (e.g. tar, zip), some platforms expect a specific packaging format.</li> <li>Digest your packages (e.g. sha256) — while we’re all enjoying a reliable internet connection and file hosting solutions — we’re not assuming it’s always the case for all users. It’s wise to verify downloads by having it digested.</li> <li>Upload to <code>&lt;provider&gt;</code> using some kind of path format scheme (e.g. to S3 as `v0.0.1/project-name/tarball).</li> <li>Set up the distribution artifact (e.g. generate a shellfile for download and verification from S3 using the given format scheme).</li> </ol> <p>For an open source project, there are many tools to use, because Github Releases and Git tags make it a breeze to construct a great release process. For other projects, Amazon S3 would be perfect for storing, and an opinionated release formatting rules helps. This is where GoReleaser shines.</p> <h2 id="using-goreleaser-with-rust">Using GoReleaser with Rust</h2> <p>Although <a href="https://github.com/goreleaser/goreleaser">GoReleaser</a> supports building just Go projects, it does so much more in the packaging and distribution department that it is extremely hard to ignore.</p> <p>It can package, generate sha256 sums, upload to Github releases or Amazon S3 or custom HTTP endpoints, as well as generate a changelog, semantic versions and a lot more. The website doesn’t show everything, so I recommend looking at <a href="https://github.com/goreleaser/goreleaser/tree/master/www/content">the repo</a> for docs and to find out what more it can support.</p> <p>So it has everything we want, but it can’t build anything other than Go.Rust cross-compilation is easy from a Linux host with <a href="https://github.com/rust-embedded/cross">cross</a>, and even easier with TravisCI with <a href="https://github.com/japaric/trust">trust</a> but it’s not easy from an OSX host, not to mention if your project is <em>not</em> open source.</p> <p><img src="https://rng0.io/blog/shipping-rust-binaries-with-goreleaser/2.png" alt="" /></p> <p>To build cross-platform with the convenience of your OSX system, this is how I do both OSX and Linux:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">docker</span></span><span class="z-meta z-function-call z-arguments z-shell"> run<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>rm</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>v</span><span class="z-meta z-group z-expansion z-command z-backticks z-shell"><span class="z-punctuation z-section z-group z-begin z-shell">`</span><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">pwd</span></span><span class="z-punctuation z-section z-group z-end z-shell">`</span></span>:/app liuchong/rustup:nightly sh<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>c</span> <span class="z-string z-quoted z-single z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&#39;</span>cd /app &amp;&amp; cargo build --release --target=x86_64-unknown-linux-gnu<span class="z-punctuation z-definition z-string z-end z-shell">&#39;</span></span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">cargo</span></span><span class="z-meta z-function-call z-arguments z-shell"> build<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>release</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>target</span><span class="z-keyword z-operator z-assignment z-option z-shell">=</span>x86_64-apple-darwin</span> </span></code></pre> <p>When you finish building, there are Linux and OSX (darwin) builds in your <code>target/</code> folder.</p> <p>For Windows builds, you might want to go back to <a href="https://github.com/japaric/trust">trust</a> and use a CI system.</p> <p>There has been efforts of making GoReleaser generic such as <a href="https://github.com/goreleaser/goreleaser/pull/520">multiple language support with Rust</a>, and some more <a href="https://github.com/goreleaser/goreleaser/issues/496">generalization discussion</a> and finally <a href="https://github.com/goreleaser/goreleaser/pull/522">a generic infrastructure</a>.</p> <p>There still is missing infrastructure to support a plug-in architecture for custom builders that would work with GoReleaser, so for now Rust still isn’t supported.</p> <h2 id="hacking-the-goreleaser-build-process">Hacking the GoReleaser Build Process</h2> <p>This hack revolves around satisfying GoReleaser by dropping a <code>dummy.go</code> file for it to build, and hooking into the <code>post</code> stage of the build.</p> <p>Here is a working <code>.goreleaser.yaml</code> set up:</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">project_name</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">cep</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">builds</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">main</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">dummy.go</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">goarch</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">amd64</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">binary</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">cep</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">hooks</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">post</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">sh build-cross-release.sh</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">archive</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">replacements</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">darwin</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">Darwin</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">linux</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">Linux</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">windows</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">Windows</span> </span><span class="z-source z-yaml"> <span class="z-constant z-numeric z-integer z-decimal z-yaml">386</span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">i386</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">amd64</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">x86_64</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">checksum</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">name_template</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-quoted z-single z-yaml"><span class="z-punctuation z-definition z-string z-begin z-yaml">&#39;</span>checksums.txt<span class="z-punctuation z-definition z-string z-end z-yaml">&#39;</span></span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">s3</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">bucket</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">cep-dev</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">acl</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">public-read</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">release</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">disable</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-constant z-language z-boolean z-yaml">true</span> </span></code></pre> <p>And the Go dummy program is trivially:</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-keyword z-other z-package z-go">package</span> <span class="z-variable z-other z-go">main</span> </span><span class="z-source z-go"> </span><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">main</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <p>Using the <code>post</code> stage we actually perform the necessary Rust builds and copying the artifacts to where GoReleaser would expect them to be, overwriting the Go artifacts (for a project named <code>cep</code>):</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-comment z-line z-number-sign z-yaml"><span class="z-punctuation z-definition z-comment z-line z-number-sign z-yaml">#</span> jondot: https://github.com/goreleaser/goreleaser/issues/962 </span></span><span class="z-source z-yaml"> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml">docker run --rm -v`pwd`:/app liuchong/rustup:nightly sh -c &#39;cd /app &amp;&amp; cargo build --release --target=x86_64-unknown-linux-gnu&#39;</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml">cargo build --release --target=x86_64-apple-darwin</span> </span><span class="z-source z-yaml"> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml">mkdir -p dist/darwin_amd64 &amp;&amp; cp target/x86_64-apple-darwin/release/cep dist/darwin_amd64</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml">mkdir -p dist/linux_amd64 &amp;&amp; cp target/x86_64-unknown-linux-gnu/release/cep dist/linux_amd64</span> </span></code></pre> <p>GoReleaser would carry on happily with the rest of the stages of the build, meaning we get everything for free!</p> <p><img src="https://rng0.io/blog/shipping-rust-binaries-with-goreleaser/3.png" alt="" /></p> <h2 id="bonus-points-godownloader">Bonus Points: GoDownloader</h2> <p>Probably not as well-known, <a href="https://github.com/goreleaser/godownloader">godownloader</a> is a sister project to GoReleaser. It will read a <code>.goreleaser.yaml</code> and generate a <code>curl-shell</code> combo for you.</p> <p>If distributing binaries via <code>curl http://your-binary.com/binary | sh</code> is your cup of tea, you can use <code>godownloader</code> and you get a clever shell script for installing your binary for free.</p> <h2 id="exploring-other-rust-tools-for-release">Exploring Other Rust Tools for Release</h2> <p>There is existing Rust work that aims to build something similar to GoReleaser, but frankly, it’s just not complete enough. You can take a look at the following:</p> <ul> <li><a href="https://crates.io/crates/cargo-release">cargo-release</a></li> <li><a href="https://crates.io/crates/cargo-deliver">cargo-deliver</a></li> <li><a href="https://crates.io/crates/cargo-hublish">cargo-hublish</a></li> </ul> <p>Finally, if you want to cross-build Rust from OSX, take a look at <a href="https://github.com/liuchong/docker-rustup">docker-rustup</a>.</p> Boost your design system with Storybook Cartesian Tue, 02 Oct 2018 00:00:00 +0000 Unknown https://rng0.io/blog/boost-your-design-system-with-storybook-cartesian/ https://rng0.io/blog/boost-your-design-system-with-storybook-cartesian/ <p><img src="https://rng0.io/blog/boost-your-design-system-with-storybook-cartesian/1.png" alt="Storybook Cartesian" /></p> <p>As part of <a href="https://github.com/jondot/awesome-designops">my investment in DesignOps</a>, I’ve bumped into a common problem with a rather nice solution.</p> <p><img src="https://rng0.io/blog/boost-your-design-system-with-storybook-cartesian/2.png" alt="" /></p> <p><a href="https://github.com/jondot/storybook-cartesian">Storybook Cartesian</a> is a library that automatically generates stories for all of your component variants and puts them back in your <a href="https://storybook.js.org">storybook</a> in a clean and tidy way.</p> <p><img src="https://rng0.io/blog/boost-your-design-system-with-storybook-cartesian/3.png" alt="" /></p> <p>Having a great React component means having it take some properties and do something useful with it. Take a look at this button:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts"><span class="z-entity z-name z-function z-ts">Button</span></span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">props</span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-parameter z-ts">button</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">style</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>{<span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">color</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-object z-ts">props</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">color</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span>} </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">enabled</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>!props.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">disabled</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-parameter z-object-binding-pattern z-ts"> <span class="z-punctuation z-definition z-binding-pattern z-object z-ts">{</span><span class="z-variable z-parameter z-ts">props</span>.<span class="z-variable z-parameter z-ts">text</span><span class="z-punctuation z-definition z-binding-pattern z-object z-ts">}</span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">button</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span><span class="z-meta z-brace z-round z-ts">)</span></span> </span></code></pre> <p>To build stories for this component in your design system, you can use some common sense and want to test enabled / disabled, and a couple of colors. But what about empty text? what about a combination of disabled and empty text? does that create anything odd?</p> <p>Instead of guessing all these combinations, you can generate them with <a href="https://github.com/jondot/storybook-cartesian">storybook-cartesian</a>.</p> <p><img src="https://rng0.io/blog/boost-your-design-system-with-storybook-cartesian/4.gif" alt="Storybook cartesian in action" /></p> <p>Storybook Cartesian in action!</p> <p>On the one hand, it’s great to be able to click through all of the variants (instead of using <a href="https://github.com/storybooks/storybook/tree/master/addons/knobs">knobs</a> to force them out manually), but more importantly, if you have these stories layed out, you can get a massive test coverage with <a href="https://github.com/storybooks/storybook/tree/master/addons/storyshots">storyshots</a>.</p> <p>If you’re impatient, there’s a fully working example in the <a href="https://github.com/jondot/storybook-cartesian/tree/master/example/app">storybook-cartesian repository</a>.</p> <h2 id="quick-start">Quick Start</h2> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn add<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>dev</span> storybook-cartesian</span> </span></code></pre> <p>To integrate in your own project, add the following to your stories in your storybook files. Here’s a real life button being exercised to its fullest:</p> <p><img src="https://rng0.io/blog/boost-your-design-system-with-storybook-cartesian/5.png" alt="" /></p> <h3 id="understanding-cartesian-stories">Understanding Cartesian Stories</h3> <p>The general structure for a <code>cartesian</code> story is this:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">cartesian</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-cast z-expr z-ts"><span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">stories</span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span><span class="z-meta z-brace z-round z-ts">)</span> </span><span class="z-source z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">add</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span><span class="z-source z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">seed</span> <span class="z-meta z-function z-expression z-ts"><span class="z-storage z-type z-function z-ts">function</span>&gt;, </span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"> <span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">title</span> <span class="z-entity z-name z-type z-ts">renderer</span><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span>, </span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"> <span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">component</span> <span class="z-entity z-name z-type z-ts">renderer</span><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span>, </span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"> <span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">valid</span> <span class="z-entity z-name z-type z-ts">combination</span> <span class="z-entity z-name z-type z-ts">filter</span> <span class="z-meta z-type z-paren z-cover z-ts"><span class="z-meta z-brace z-round z-ts">(</span><span class="z-entity z-name z-type z-ts">optional</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span>, </span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"> <span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">story</span> <span class="z-entity z-name z-type z-ts">apply</span> <span class="z-entity z-name z-type z-ts">function</span> <span class="z-meta z-type z-paren z-cover z-ts"><span class="z-meta z-brace z-round z-ts">(</span><span class="z-entity z-name z-type z-ts">optional</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"> ) </span></span></code></pre> <p>Which gets you this kind of story layout <em>generated automatically</em> (for now the last “All/all variants” is a story discussed:</p> <p><img src="https://rng0.io/blog/boost-your-design-system-with-storybook-cartesian/6.png" alt="Stories layout" /></p> <p>Automatically generated with storybook-cartesian</p> <p>Your <code>seed</code> function is responsible to generate content in the form of:</p> <p><img src="https://rng0.io/blog/boost-your-design-system-with-storybook-cartesian/7.png" alt="" /></p> <p>Your <code>titleRender</code> function gets an instance of your props and returns a string. This is where you get to build those fancy story titles that will make every generated story super visible.</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts"><span class="z-entity z-name z-function z-ts">titleRender</span></span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">props</span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span><span class="z-string z-template z-ts"><span class="z-punctuation z-definition z-string z-template z-begin z-ts">`</span><span class="z-meta z-template z-expression z-ts"><span class="z-punctuation z-definition z-template-expression z-begin z-ts">${</span></span><span class="z-meta z-template z-expression z-ts"><span class="z-meta z-embedded z-line z-ts"><span class="z-variable z-other z-object z-ts">props</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">one</span></span><span class="z-punctuation z-definition z-template-expression z-end z-ts">}</span></span> / <span class="z-meta z-template z-expression z-ts"><span class="z-punctuation z-definition z-template-expression z-begin z-ts">${</span></span><span class="z-meta z-template z-expression z-ts"><span class="z-meta z-embedded z-line z-ts"><span class="z-variable z-other z-object z-ts">props</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">check</span></span><span class="z-punctuation z-definition z-template-expression z-end z-ts">}</span></span><span class="z-punctuation z-definition z-string z-template z-end z-ts">`</span></span></span> </span></code></pre> <p>Your <code>storyRender</code> function gets an instance of your props and returns a component. This is where you can just spread the <code>props</code> out and have an easy win, or do something fancy and build out a storyfied component proper.</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts"><span class="z-entity z-name z-function z-ts">componentRender</span></span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">props</span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span><span class="z-cast z-expr z-ts"> <span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">Button</span> <span class="z-meta z-object z-type z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-keyword z-operator z-spread z-ts">...</span><span class="z-entity z-name z-type z-ts">props</span><span class="z-punctuation z-definition z-block z-ts">}</span></span> /<span class="z-meta z-brace z-angle z-ts">&gt;</span></span></span></span> </span></code></pre> <p>And to make use of all of these with <code>cartesian</code> we can now do:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">cartesian</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">storiesOf</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>Button/Cartesian<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span> </span><span class="z-source z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">add</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">seedfn</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">titleRender</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">componentRender</span> </span><span class="z-source z-ts"> <span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <p>Last thing to remember — your <code>seedfn</code> is a function. Currently it just returns a static bag of props, but it’s a function for good reason. You can use tools like <a href="https://github.com/typicode/react-fake-props">react-fake-props</a> to generate this seed bag of props automatically as well — without writing any verbatim seed code. In addition you can use this function to fetch content dynamically from a backend.</p> <h3 id="validation-and-advanced-rendering">Validation and Advanced Rendering</h3> <p>Some times not all prop combinations make sense. For example if you have an <code>isLoading</code> and a <code>results</code> props it doesn't make sense to have both <code>true</code> and <code>results</code> populated:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> doesn&#39;t make sense</span> </span><span class="z-source z-ts"><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">SearchResults</span> <span class="z-variable z-other z-readwrite z-ts">isLoading</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">true</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-variable z-other z-readwrite z-ts">results</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>hello<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>world<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-square z-ts">]</span></span>}&gt; </span></span></span></span></code></pre> <p>For this, we have a <code>valid</code> function that we can add on top of what we’ve seen until now. With it, the following will filter out this invalid combination:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">cartesian</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">storiesOf</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>Button/Cartesian<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span> </span><span class="z-source z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">add</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">seedfn</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">titleRender</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">componentRender</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span><span class="z-source z-ts"> <span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">props</span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span><span class="z-keyword z-operator z-logical z-ts">!</span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-object z-ts">props</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">isLoading</span> <span class="z-keyword z-operator z-logical z-ts">&amp;&amp;</span> <span class="z-variable z-other z-object z-ts">props</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">results</span><span class="z-meta z-brace z-round z-ts">)</span> </span><span class="z-source z-ts"> <span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <p>The default <code>valid</code> function, if not given, is simply returning <code>true</code> always.</p> <p>Some other times you might want to customize how stories get created. By default, <code>react-cartesian</code> will add one story per variation. For some components types (such as a button) you might not want to “spend” an entire story.</p> <p>For example, let’s say you want just one story to <em>contain all</em> cartesian product items.</p> <p><img src="https://rng0.io/blog/boost-your-design-system-with-storybook-cartesian/8.png" alt="All variants" /></p> <p>For this, we have another optional function that we can use, that gives you <code>stories</code> and <code>candidates</code> back with which you can do what ever you want:</p> <p><img src="https://rng0.io/blog/boost-your-design-system-with-storybook-cartesian/9.png" alt="" /></p> <h2 id="try-it-out">Try it out!</h2> <p>Storybook Cartesian is a great way to get your design system going. It will allow you to spread out all these UI states that you haven’t considered with no effort at all. You can take a look <a href="https://github.com/jondot/storybook-cartesian">on Github</a> and you’re welcome to try it out (or submit pull requests!).</p> <p>Have fun!</p> Build a Free Electron Analytics Stack with Google Analytics and Redux Fri, 28 Sep 2018 00:00:00 +0000 Unknown https://rng0.io/blog/build-a-free-electron-analytics-stack-with-google-analytics-and-redux/ https://rng0.io/blog/build-a-free-electron-analytics-stack-with-google-analytics-and-redux/ <p><img src="https://rng0.io/blog/build-a-free-electron-analytics-stack-with-google-analytics-and-redux/1.png" alt="" /></p> <p>If you’re looking here, there’s a good chance you already realized that there aren’t many options for wiring your Electron app with analytics.</p> <p>Currently, there’s nothing great like <a href="https://firebase.google.com/docs/analytics/">Firebase Analytics</a> for Electron. An in fact, being that Electron is a desktop app, I don’t see a great reason why there shouldn’t be. In many ways Electron apps are similar to mobile apps in that regard and so lend themselves well to getting a ton of value from analytics integration. <em>PS — there’s</em> <a href="https://github.com/getsentry/sentry-electron"><em>Sentry</em></a> <em>integration for Electron, but it’s not analytics, and I found it flaky / buggy so far for some use cases</em>.</p> <p><img src="https://rng0.io/blog/build-a-free-electron-analytics-stack-with-google-analytics-and-redux/2.png" alt="" /></p> <p>Since Electron apps are Desktop apps and not websites, there’s a good chance your users would be using your app without an internet connection or with an intermittent connection. For that reason any solution you choose should be robust and take that into account. What’s more, if you assumed you’ll just add the usual Javascript universal analyics tag to your Electron main HTML page, you’re in for a surprise; not only will you lose events once your app goes offline, but Google Analytics will also:</p> <ul> <li>Require a real <code>http</code> protocol behind your app, which isn't what's there. Your app is behind a <code>file://</code> protocol and so Google Analytics will fail silently (unless debug variant of Google Analytics is used and tracing enabled)</li> <li>Try to use cookies, and fail silently again</li> <li>Have nothing to work with in terms of identifying pages</li> <li>Work outside of <code>fetch</code> and so you won't see anything in the Network tab in your Electron devtools</li> </ul> <h2 id="redux-beacon-electron">redux-beacon-electron</h2> <p>If you have an app powered by Redux, you’re already in a perfect state to integrate analytics. In fact you can see every <em>action</em> in your app architecture as an analytics event, and since actions in Redux are granular and isolated, it’s extremely simple to turn them into analytics event.</p> <p><img src="https://rng0.io/blog/build-a-free-electron-analytics-stack-with-google-analytics-and-redux/3.gif" alt="" /></p> <p><a href="https://github.com/rangle/redux-beacon">Redux Beacon</a> is such infrastructure, that hooks into Redux and lets you map actions to analytics events as one-to-one or even fan-out (report to many analytics providers). You can use one of many ready-made integrations including Google Analytics and <a href="https://segment.io">Segment.io</a> and build your own.</p> <p><code>redux-beacon-electron</code> is a library that includes a customized Electron-friendly Google Analytics target, an opinionated Redux Beacon event mapper and an action tracker that will help you build a simple and maintainable analytics solution.</p> <h2 id="quick-start">Quick Start</h2> <p>Set up your <a href="https://github.com/rangle/redux-beacon">Redux Beacon</a> infrastructure as you would normally. Then, install <code>redux-beacon-electron</code>:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn add redux-beacon-electron</span> </span></code></pre> <p>Integrate your Electron app with Redux Beacon and <code>redux-beacon-electron</code></p> <p><img src="https://rng0.io/blog/build-a-free-electron-analytics-stack-with-google-analytics-and-redux/4.png" alt="" /></p> <p>You can provide your UA tracking identifier verbatim or pull it from configuration — initialization is done on the spot (and not through injecting the generic GA snippet into the DOM like many libraries do), and even more — you can initialize as many targets as you like (!), so if you want to report different areas of your app to different accounts you most definitely can.</p> <p>Then, somewhere in your Redux actions (I’m using <code>redux-actions</code> but it doesn't really matter if you use something else):</p> <p><img src="https://rng0.io/blog/build-a-free-electron-analytics-stack-with-google-analytics-and-redux/5.png" alt="" /></p> <p>The <code>onMounted</code> action will generate a page view that's no different to Google Analytics than any other website page view, which is what we want in order to get these events through (otherwise Google Analytics will be happy to drop non-conforming events silently). It will do the smart thing regarding page name, based on your Redux event - in this case it will split (based on "/") the action type and use "settings" as the page name.</p> <p>The <code>setMuteState</code> action will emit an <code>event</code> hit to Google Analytics, and it too will break up the namespace into an "app" category and a "SET <em>MUTE</em> STATE" action.</p> <h3 id="meta-events-mapper">Meta Events Mapper</h3> <p>You can use the <code>actionMetaEventMapper</code> if you believe (as I do) that actions should say how they map themselves to analytics events (as opposed to having a big switch case or mapping dictionary in a centralized place).</p> <p>This means that the form of a <em>Flux Standard Action</em> that is tracked is now:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">type</span><span class="z-punctuation z-separator z-label z-ts">:</span><span class="z-keyword z-operator z-spread z-ts">...</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">action</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-keyword z-operator z-spread z-ts">...</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">meta</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">track</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">action</span></span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">hit</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>event<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-keyword z-operator z-bitwise z-ts">|</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>pageview<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">category</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">action</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">label</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">value</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <h3 id="one-last-thing">One Last Thing</h3> <p>A nice side effect to this approach is that you have end-to-end debuggability: Redux action -&gt; Event -&gt; network fetch call.</p> <p>Actions and events:</p> <p><img src="https://rng0.io/blog/build-a-free-electron-analytics-stack-with-google-analytics-and-redux/6.png" alt="" /></p> <p>Network:</p> <p><img src="https://rng0.io/blog/build-a-free-electron-analytics-stack-with-google-analytics-and-redux/7.png" alt="" /></p> <p><em>For more information and pull requests see</em> <a href="https://github.com/jondot/redux-beacon-electron"><em>https://github.com/jondot/redux-beacon-electron</em></a></p> Four Upcoming Javascript Features You Didn’t Know Exists Tue, 04 Sep 2018 00:00:00 +0000 Unknown https://rng0.io/blog/four-upcoming-javascript-features-you-didnt-know-exists/ https://rng0.io/blog/four-upcoming-javascript-features-you-didnt-know-exists/ <p><img src="https://rng0.io/blog/four-upcoming-javascript-features-you-didnt-know-exists/1.png" alt="" /></p> <p><a href="https://babeljs.io/blog/2018/08/27/7.0.0">Babel 7.0 was released this week</a> summing up two years of work and 4000 commits (yes, that’s 4000!).</p> <p>There’s no better time to take a look at what the next version of Javascript will look like, and how you can use it <em>today</em> (if you’re impatient, jump to the bottom).</p> <p>Have fun!</p> <h2 id="1-pipeline-operator">1. Pipeline Operator</h2> <p><img src="https://rng0.io/blog/four-upcoming-javascript-features-you-didnt-know-exists/2.png" alt="" /></p> <p>One of the upcoming features I’m excited about is the <a href="https://babeljs.io/docs/en/next/babel-plugin-proposal-pipeline-operator">pipeline operator</a>. With this, Javascript is taking a big step towards functional programming, and responds to the “new gen” community that already is neck deep in functional programming — just look at <a href="https://ramdajs.com/">Ramda</a>, <a href="https://github.com/fantasyland/fantasy-land">fantasyland</a>, and functional-reactive programming libraries like <a href="https://github.com/cujojs/most">most</a> to get an intuition for what the future holds.</p> <p>The pipeline operator roots itself in programming languages like F# and <a href="https://elixirschool.com/en/lessons/basics/pipe-operator/">Elixir</a>, and is majorly used in scenarios where <a href="https://rng0.io/blog/functional-programming-with-python-for-people-without-time/">you treat your code as a data pipeline</a>, so you can do this:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts"><span class="z-entity z-name z-function z-ts">mul</span></span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">x</span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span><span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">y</span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span><span class="z-variable z-other z-readwrite z-ts">x</span><span class="z-keyword z-operator z-arithmetic z-ts">*</span><span class="z-variable z-other z-readwrite z-ts">y</span></span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">result</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-numeric z-decimal z-ts">42</span> <span class="z-keyword z-operator z-bitwise z-ts">|</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">mul</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-constant z-numeric z-decimal z-ts">2</span><span class="z-meta z-brace z-round z-ts">)</span></span> </span><span class="z-source z-ts"> <span class="z-keyword z-operator z-bitwise z-ts">|</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> <span class="z-support z-class z-console z-ts">console</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-console z-ts">log</span> </span></code></pre> <p>Which “pipes” the number 42 through a multiplication and a log operation. Note that <code>mul</code> is a multiplication function in its <a href="https://en.wikipedia.org/wiki/Currying">curried</a> form. Currying as a concept fits extremely well when we're dealing with functional programming (and now you can see why).</p> <p>If you’re not using Babel (say you’re on Typescript) you can still have a similar experience today with Ramda’s <a href="https://ramdajs.com/docs/#pipe">pipe</a> function or the Lodash <a href="https://lodash.com/docs/4.17.10#flow">flow</a> function, or if you prefer Pyton then the toolz <a href="https://toolz.readthedocs.io/en/latest/api.html#toolz.functoolz.pipe">pipe</a> function.</p> <h2 id="2-optional-chaining">2. Optional Chaining</h2> <p><img src="https://rng0.io/blog/four-upcoming-javascript-features-you-didnt-know-exists/3.png" alt="" /></p> <p>The <a href="https://babeljs.io/docs/en/next/babel-plugin-proposal-optional-chaining">optional chaining operator</a> (‘?’) tucked onto properties will make chaining access not explode, and will carry on an ‘undefined’ value if found, up to the last property access.</p> <p>This makes it easy to perform the same operation without any edge case logic, and avoid bad input</p> <p><a href="https://babeljs.io/docs/en/next/babel-plugin-proposal-optional-chaining">Optional chaining</a> is one of the solutions for the “null pointer exception” problem — a pain that has a long history in programming languages, even at one point named “ <a href="https://en.wikipedia.org/wiki/Tony_Hoare#Apologies_and_retractions">The Billion Dollar Mistake</a>” — is the reputation the <em>null</em> value got.</p> <p>In fact if <em>null</em> was not invented, we would probably be looking at concepts that are emerging into popular domain nowadays, like the <a href="https://www.oracle.com/technetwork/articles/java/java8-optional-2175753.html">Optional</a> type, or the <a href="http://hackage.haskell.org/package/base-4.11.1.0/docs/Data-Maybe.html">Just/Maybe</a> types, or first-class error handling in <a href="https://blog.golang.org/error-handling-and-go">Go</a>, or first-class syntax support in <a href="https://kotlinlang.org/docs/reference/null-safety.html">Kotlin</a>, <a href="https://docs.swift.org/swift-book/LanguageGuide/OptionalChaining.html">Swift</a> and more — and now we have it in Javascript!.</p> <h2 id="3-nullish-coalescing">3. Nullish Coalescing</h2> <p><img src="https://rng0.io/blog/four-upcoming-javascript-features-you-didnt-know-exists/4.png" alt="" /></p> <p>The nullish coalescing operator (‘??’) lets you provide a default value if left-hand is nullish (undefined, null, etc.) and so compared to the previous example will never return undefined values.</p> <p>I used to use the null coalescing operator <a href="https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/operators/null-coalescing-operator">in C#</a> years ago, and when I was doing Scala I used to add it as a custom operator as well, I’m expecting this to go mainstream in Javascript very quickly.</p> <h2 id="4-logical-assignment">4. Logical Assignment</h2> <p><img src="https://rng0.io/blog/four-upcoming-javascript-features-you-didnt-know-exists/5.png" alt="" /></p> <p>The <a href="https://babeljs.io/docs/en/next/babel-plugin-proposal-logical-assignment-operators.html">logical assignment operator</a> (‘||=’, ‘&amp;&amp;=’), inspired by Ruby and CoffeeScript sprinkles a little syntax sugar for when you want default values semantics but with a concise form of code.</p> <p>Granted, for some this semantic matter more, and for some it matters less. For those who’s doing immutable code (functional programming) it will matter less.</p> <h3 id="one-more-thing">One More Thing…</h3> <p>There are <strong>two more features</strong> in the baking for next generation Javascript.</p> <p>The upcoming <a href="https://github.com/tc39/proposal-pattern-matching">pattern matching proposal</a> is the one I’m <em>the most excited about</em> and been following for around a year now.</p> <p>The next one I’m excited about is the <a href="https://github.com/tc39/proposal-partial-application">partial application</a> proposal which means you can bake half functions (curry) to be used in pipelines very easily.</p> <h2 id="show-me-the-code">Show Me the Code</h2> <p>You can play with all of these today by cloning the <a href="https://github.com/jondot/babel-pipeline-etc">Github repo here</a>.</p> <p>If you want to use the new features today in production, you can use <a href="https://github.com/jondot/goddamn-javascript-babel">my babel mega-configuration repo</a> which make all of Babel’s configuration seamless and transparent for you.</p> Functional Programming with Python for People Without Time Wed, 22 Aug 2018 00:00:00 +0000 Unknown https://rng0.io/blog/functional-programming-with-python-for-people-without-time/ https://rng0.io/blog/functional-programming-with-python-for-people-without-time/ <p><img src="https://rng0.io/blog/functional-programming-with-python-for-people-without-time/1.png" alt="" /></p> <p>Functional programming for the purpose of this article (and I claim — for the purpose of most of your day to day work), is all about building pipelines and moving data through those pipelines.</p> <p>Two constructs that you probably already heard of make up much of our pipeline oriented work: these are <code>map</code> and <code>reduce</code>.</p> <p>As an appetizer, here’s a puzzle — pick all the first names from a list of people, and print them joined by a comma (“,”). Make the resulting code clear to read and as minimal as possible. Let your friends try, too.</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">people</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span> </span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-meta z-mapping-or-set z-python"><span class="z-punctuation z-section z-mapping-or-set z-begin z-python">{</span> </span></span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping-or-set z-python"></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">first_name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Bruce<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">last_name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Wayne<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span> </span></span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-value z-python"></span><span class="z-meta z-mapping z-python"> <span class="z-punctuation z-section z-mapping z-end z-python">}</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-meta z-mapping-or-set z-python"><span class="z-punctuation z-section z-mapping-or-set z-begin z-python">{</span> </span></span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping-or-set z-python"></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">first_name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Joker<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">last_name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span> </span></span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-value z-python"></span><span class="z-meta z-mapping z-python"> <span class="z-punctuation z-section z-mapping z-end z-python">}</span></span> </span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span> </span></code></pre> <p>Also, take a moment and try it imperatively.</p> <p>Let’s do this with <code>map</code> and <code>reduce</code> only.</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">first_names</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">map</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">person</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">person</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">first_name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">people</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">joined_names</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">reduce</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">acc</span><span class="z-punctuation z-separator z-parameters z-python">,</span> <span class="z-variable z-parameter z-python">name</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">acc</span></span> <span class="z-keyword z-operator z-arithmetic z-python">+</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">,<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span> <span class="z-keyword z-operator z-arithmetic z-python">+</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">name</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">first_names</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-constant z-other z-placeholder z-python"><span class="z-punctuation z-definition z-placeholder z-begin z-python">{</span><span class="z-punctuation z-definition z-placeholder z-end z-python">}</span></span><span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">format</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">joined_names</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Bruce,Joker </span></span></code></pre> <p>Let’s get rid of ephemeral variable <code>first_names</code> and <code>joined_names</code> which just stand there and do nothing but serve as a fancy temporary bag of values ( <a href="http://www.sicpdistilled.com/section/1.1.5/">by substitution</a>):</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">joined_names</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">reduce</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">acc</span><span class="z-punctuation z-separator z-parameters z-python">,</span> <span class="z-variable z-parameter z-python">name</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">acc</span></span> <span class="z-keyword z-operator z-arithmetic z-python">+</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">,<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span> <span class="z-keyword z-operator z-arithmetic z-python">+</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">name</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">map</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">person</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">person</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">first_name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">people</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-constant z-other z-placeholder z-python"><span class="z-punctuation z-definition z-placeholder z-begin z-python">{</span><span class="z-punctuation z-definition z-placeholder z-end z-python">}</span></span><span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">format</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">joined_names</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Bruce,Joker </span></span></code></pre> <p>And once again:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-constant z-other z-placeholder z-python"><span class="z-punctuation z-definition z-placeholder z-begin z-python">{</span><span class="z-punctuation z-definition z-placeholder z-end z-python">}</span></span><span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">format</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">reduce</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">acc</span><span class="z-punctuation z-separator z-parameters z-python">,</span> <span class="z-variable z-parameter z-python">name</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">acc</span></span> <span class="z-keyword z-operator z-arithmetic z-python">+</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">,<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span> <span class="z-keyword z-operator z-arithmetic z-python">+</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">name</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">map</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">person</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">person</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">first_name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">people</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Bruce,Joker </span></span></code></pre> <p>Super hard to read, and you’re already making the conclusion that functional programming goes against all craftsmanship principles in software, that it’s not maintainable and that all functional code lives somewhere in the academy in one of their ivory towers.</p> <p>Well, it’s not true; and this is the point where most people fall off, marks “never go there again” and “functional programming is for stuckups”.</p> <p>Let’s reverse engineer this process. Taking a couple steps back, we can see that we have the concepts of a <em>first name extractor</em>:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">person</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">person</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">first_name</span></span></span> </span></code></pre> <p>And we have a kind of a <em>joiner of things</em>:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">acc</span><span class="z-punctuation z-separator z-parameters z-python">,</span> <span class="z-variable z-parameter z-python">name</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">acc</span></span> <span class="z-keyword z-operator z-arithmetic z-python">+</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">,<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">name</span></span></span> </span></code></pre> <p>But we can make these generic, because there’s nothing special about a comma being a <em>separator</em> and a first name <em>field</em>:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">joiner</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">sep</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">acc</span><span class="z-punctuation z-separator z-parameters z-python">,</span> <span class="z-variable z-parameter z-python">name</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">acc</span></span> <span class="z-keyword z-operator z-arithmetic z-python">+</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">sep</span></span> <span class="z-keyword z-operator z-arithmetic z-python">+</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">name</span></span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">extractor</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">field</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">person</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">person</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">field</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span></span> </span></code></pre> <p>If you’re thinking — excessive use of <code>lambdas</code> - that's the Pythonic <a href="https://news.ycombinator.com/item?id=14594872">anti-functional-language mindset</a> rearing its head, hopefully we'll break through that soon. Any way, refactoring to incorporate our latest moves, we now come at this:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-constant z-other z-placeholder z-python"><span class="z-punctuation z-definition z-placeholder z-begin z-python">{</span><span class="z-punctuation z-definition z-placeholder z-end z-python">}</span></span><span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">format</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">reduce</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">joiner</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">,<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">map</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">extractor</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">first_name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">people</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Bruce,Joker </span></span></code></pre> <p>You read this inside out, right? We first look at <code>print</code>, then <code>format</code>, then <code>reduce</code>, and realize that we don't know enough, and dig down through the parens until we hit the core - <code>extractor</code>, backed by <code>map</code> (keep in mind <code>extractor</code> is invoked on <code>'first_name'</code> but then returns a <em>function</em> which <code>map</code> is happy to take).</p> <p>We understand that we <code>map</code> over people, giving back a new list, which <code>reduce</code> takes, and joins. While this is fun as a puzzle, it can be tricky to read once we add some more <code>map</code>/ <code>reduce</code> steps.</p> <p>To deal with these problems generically, we need to build a <em>pipeline</em> of functions that go over our <em>data</em> — or using the correct analogy — a <em>pipeline</em> through which our data flows. This is one of the areas where functional programming shines.</p> <p>Here’s how we solve that puzzle properly:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-constant z-other z-placeholder z-python"><span class="z-punctuation z-definition z-placeholder z-begin z-python">{</span><span class="z-punctuation z-definition z-placeholder z-end z-python">}</span></span><span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">format</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">pipe</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">people</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">map</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">get_in</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">first_name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">reduce</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">joiner</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">,<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p>Our pipe is built out of functions ( <code>pipe</code>, <code>map</code>, <code>reduce</code>) that in turn are built out of yet more functions ( <code>get_in</code>, <code>join</code>), and then some primitives ( <code>['first_name']</code>, <code>","</code>).</p> <p>There’s <em>nothing</em> in here that forces you to go to a special object oriented programming school for. No inheritence, metaclasses, metaprogramming, abstract classes, mixins, and other friends.</p> <p>Just functions and primitives.</p> <p>In addition, this code is parallelizeable by default (just swap <code>map</code> with a <a href="https://toolz.readthedocs.io/en/latest/parallelism.html">pmap</a>), have zero side-effects or internal state (no variables that it maintains, so less bugs), and is testable with <a href="https://github.com/syrusakbary/snapshottest">snapshots</a> (give it input, expect output). And that means no concurrency problems or race conditions and mocking and test rig school to go to either.</p> <p>The functions <code>get_in</code>, <code>join</code>, and <code>pipe</code> come directly from <a href="https://toolz.readthedocs.io/en/latest/api.html">toolz</a>, a functional programming toolkit for Python. <code>map</code> and <code>reduce</code> come from toolz as well (they come standard from Python too), but notice everything comes in its <a href="https://en.wikipedia.org/wiki/Currying"><em>curried</em></a> form.</p> <p>Which means that when we made our <em>extractor</em> and <em>joiner</em>, we received some “set up” variables, such as a separator (in the case of <em>joiner</em>) and a field to extract (in the case of <em>extractor</em>) and returned <em>functions</em> that were “configured” to perform with these variables.</p> <p>In the more generalized sense, we created a <em>curried</em> version of the functions we used, and if we want to save some manual work it looks like this, using <code>@curry</code> from toolz:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-annotation z-python"><span class="z-punctuation z-definition z-annotation z-python">@</span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-annotation z-python"><span class="z-meta z-generic-name z-python">curry</span></span></span></span> </span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">extractor</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">field</span><span class="z-punctuation z-separator z-parameters z-python">,</span> <span class="z-variable z-parameter z-python">person</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">person</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">field</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span> </span></code></pre> <p>Which behaves like this:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">extract_with_field</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">extractor</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">first_name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> returns a _function_ that takes a person </span></span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> and will always extract based on the field &#39;first_name&#39; </span></span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> (because it has it in its tummy) </span></span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">extract_with_field</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-section z-mapping z-begin z-python">{</span></span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">first_name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Bruce<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">last_name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Wayne<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-section z-mapping z-end z-python">}</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> &#39;Bruce&#39; </span></span></code></pre> <p>In this case <em>currying</em> allows us to <em>capture</em> (or cache) a behavior of a function like <em>extractor</em> at different times for different purposes; or more specifically extracting different fields:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">get_first_name</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">extractor</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">first_name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">get_last_name</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">extractor</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">last_name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p>Some would say we built a couple <em>accessors</em> up there.</p> <p>Let’s come to a full stop here. How different is all of this from object oriented programming?</p> <p>Let’s see how we would have built our <code>extractor</code> - for illustrative purposes we'll use ES6 (Using Javascript for OOP code just to be more strict and <em>completely</em> separate the different approaches for the reader’s convenience):</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-storage z-type z-class z-ts">class</span> <span class="z-entity z-name z-type z-class z-ts">Extractor</span> <span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-storage z-type z-ts">constructor</span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">field</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">field</span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-readwrite z-ts">field</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">extract</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">person</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-readwrite z-ts">person</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">field</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p>And to invoke it:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-variable z-other z-readwrite z-ts">extractor</span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-new z-expr z-ts"><span class="z-keyword z-operator z-new z-ts">new</span> <span class="z-entity z-name z-type z-ts">Extractor</span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>first_name<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span></span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">people</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">map</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">person</span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">extractor</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">extract</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">person</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <p>If you’re thinking — it’s not <em>too</em> bad; at least, not too impressive to call in a whole army of functional programming operators and compose those in the voodo that is <code>pipe</code> s and <code>map</code> s and currying - then you would be right.</p> <p>This Extractor thing is the <a href="https://en.wikipedia.org/wiki/Command_pattern">command pattern</a> in disguise. Here is Extractor again as a <em>command</em>, with the standard <code>execute</code> method coming from an interface being overrided:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-storage z-type z-class z-ts">class</span> <span class="z-entity z-name z-type z-class z-ts">Extractor</span> <span class="z-storage z-modifier z-ts">extends</span> <span class="z-entity z-other z-inherited-class z-ts">Command</span> <span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-storage z-type z-ts">constructor</span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">field</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">field</span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-readwrite z-ts">field</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">execute</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">person</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-readwrite z-ts">person</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">field</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p>And now if we have a list of commands, we can proudly declare that don’t care <em>what</em> they are, but we only care what they <em>do</em>:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-variable z-other z-readwrite z-ts">cmds</span> <span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span> </span></span><span class="z-source z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-variable z-other z-readwrite z-ts">extract</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-variable z-other z-readwrite z-ts">join</span> </span></span><span class="z-source z-ts"><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">]</span></span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">cmds</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">each</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">cmd</span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">cmd</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">execute</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">subject</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <p>Behold, a pipeline. We can build pipelines in object oriented programming! and it takes away the edge from the “everything is a pipeline and functional programming is solving pipelines” argument we made earlier, yay! nothing new to learn!.</p> <h2 id="cracks-in-the-ice">Cracks in the Ice</h2> <p>We ended the previous part with stating that with a good measure of abstraction, functional programming doesn’t offer a considerable advantage over the “traditional” way of design, object oriented. It’s a lie.</p> <p>Let’s start with this. With object oriented design we can abstract and build everything that functional programming offers. In fact, taking our command pattern example into account, that’s how Java’s implements <a href="https://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexpressions.html">lambda expressions</a>.</p> <p>That’s a particularily good example, because Java is known for its never-break-compatibility approach; it means that what ever Java is going to do to add in <em>lambdas</em>, they are going to use something that’s already there — and they did — anonymous <em>classes</em> were what backed lambdas in Java.</p> <p>But here’s where this crystal-clear-perfect argument cracks. In our pipeline example above with our Executors — how do you feed in the output of one executor as the input for the next one? well, you have to build that infrastructure.</p> <p>With functional programming, those <em>abstractions</em> are not abstractions that you have to custom build. They’re part of the language, mindset, and ecosystem. Generically speaking — it’s all about <em>impedence mismatch</em> and <em>leaky abstractions</em> and when it comes to data and functions; there’s no mismatch because it’s built up from the core.</p> <p>The thesis is — that to build a functional programming approach over a an object-oriented playground — is going to crash and burn at one point or another: be it bad modeling of abstractions, performance problems, bad developer ergonomics, and the worst — wrong mindset.</p> <h3 id="minimal-surface-areas">Minimal Surface Areas</h3> <p>Being able to model problems and solutions in a functional way, transcends above traditional abstraction; the object-oriented approach, in comparison, is crude, inefficient and prone to maintenance problems.</p> <p>To understand why, let’s answer the following. What are the chances for this class to fail?</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">class Billing { </span><span class="z-text z-plain"> compute_address() </span><span class="z-text z-plain"> billing_address() </span><span class="z-text z-plain"> contact_info(user) </span><span class="z-text z-plain"> transaction_holder(holder_id) </span><span class="z-text z-plain"> as_string() </span><span class="z-text z-plain"> save() </span><span class="z-text z-plain"> update() </span><span class="z-text z-plain"> get(id) </span><span class="z-text z-plain">} </span></code></pre> <p>Assuming we have all of the past error data in our Splunks and JIRAs, let’s pour everything onto a spreadsheet, and assign a probability to fail computed based on the past year to each method:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">class Billing { </span><span class="z-text z-plain"> compute_address() </span><span class="z-text z-plain"> // 0.01 (someone had a typo) </span><span class="z-text z-plain"> </span><span class="z-text z-plain"> billing_address() </span><span class="z-text z-plain"> // 0.01 (someone got the wrong address) </span><span class="z-text z-plain"> </span><span class="z-text z-plain"> contact_info(user) </span><span class="z-text z-plain"> // 0.05 (user has a 0.02 failure rate + computation in contact_info) </span><span class="z-text z-plain"> </span><span class="z-text z-plain"> transaction_holder(holder_id) </span><span class="z-text z-plain"> // 0.08 (does I/O) </span><span class="z-text z-plain"> </span><span class="z-text z-plain"> as_string() </span><span class="z-text z-plain"> // 0.01 (someone misrepresented it) </span><span class="z-text z-plain"> </span><span class="z-text z-plain"> save() </span><span class="z-text z-plain"> // 0.08 (does I/O) </span><span class="z-text z-plain"> </span><span class="z-text z-plain"> update() </span><span class="z-text z-plain"> // 0.08 (does I/O) </span><span class="z-text z-plain"> </span><span class="z-text z-plain"> get(id) </span><span class="z-text z-plain"> // 0.08 (does I/O) </span><span class="z-text z-plain">} </span></code></pre> <p>There’s a 40% chance that this code will fail at some point in the next year (for the nitpickers — assuming this code runs independently on different machines, in parallel, of course — and we’re showing just part of the codebase where all probabilities ultimately end up as 1.0):</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">(p(a) or p(b) = p(a+b)) </span><span class="z-text z-plain">0.08 + 0.08 + 0.08 + 0.01 + 0.08 + 0.05 + 0.01 + 0.01 </span><span class="z-text z-plain">-&gt; 0.4 </span></code></pre> <p>While the numbers are not important (and they’re not real, remember?), nor the story of a Billing class that failed at this and that probabilities over the last year, what is important is the fact that — <em>the more methods we had, the more chances we had to fail</em>.</p> <p>And not only that: <em>any class that interfaces with Billing is mating its own failure rate with that dreadful number of 0.4</em>. This is the concept of how large interfaces make up a large surface area to commit against which makes things brittle.</p> <p>The solution is obvious, if I’m a <code>User</code> class and I need to interface with <code>Billing</code> I want to take a <em>subset</em> of <code>Billing</code>'s interface. Maybe only that subset that is <em>stable</em>?</p> <p>Functional programming educate for minimalistic, compact ideas, that will push you to create small area interfaces, and so you’ll typically have many small concepts rather than a big-design-upfront abstractions in form of objects and hierarchies.</p> <h2 id="finding-your-path-in-a-functional-world">Finding Your Path in a Functional World</h2> <p>If you tried taking up functional programming on your own, you probably know this feeling: there seems to be something everyone knows but you don’t. There’s some kind of alchemy of functions and popular operators (such as <code>pipe</code>, <code>lift</code>, and <code>mapcat</code>) and types (such as <code>Maybe</code>) that everyone seems to "get" but you don't.</p> <p>What you need to remember is: for now — that’s OK. It’s OK because we’re going to map the reasons for why it happens (that’s half the battle!).</p> <h3 id="it-s-got-math-roots">It’s Got Math Roots</h3> <p>There’s a <em>ton</em> of mathematically-oriented theories around it, an amount fifty years of research have allowed to accumulate. There are plenty of theories and proofs from mathematics applied directly to functional programming and it’s not so easy to digest without an appropriate time investment <em>both</em> in the history of such an academic topic <em>and</em> in its current form. <a href="https://en.wikipedia.org/wiki/Category_theory">Category theory</a> is one interesting topic; and it’s not trivial to catch up with if all you’re interested in is building software.</p> <h3 id="it-s-language-abstraction">It’s Language + Abstraction</h3> <p>There’s actually <em>two</em> major schools of thought in the functional programming world — the ML statically-typed-backed-by-category-theory-and-fancy-type-systems school and the LISP dynamically-typed-code-is-data-we’re-in-the-60s-so-lighten-up school. There’s overlap in some constructs (e.g. we’ve seen map, reduce) but a strong difference in mindset (types vs no types).</p> <p>The popular claim is, like in the OOP world, that strong types — especially in functional languages — are critical for large codebase maintenance since there are so many small functions. The <em>difference</em> between OOP types and FP types is that the latter are extremely thought out and that the same core types and thinking will follow you in most ML-family language.</p> <p>In other words, it’s like saying that when OOP was born, it was also born with the Gang-of-Four design patterns baked into it’s core as its backing theory of thought (outside of types and inheritence and methods etc.), and every implemented OOP language included these patterns and abstractions by default for you to take advantage of, and that these patterns were bullet-proofed by centuries of research. But that can already never be correct — the Singleton pattern is by now widely recognized as an anti-pattern, and GoF authors said they would remove it, if only they could go back in time.</p> <h3 id="that-purity-thing">That Purity Thing</h3> <p>There are largely two approaches a functional language need to pick from. These are the “pure” — no side-effects, and the “impure” (or maybe “pragmatic”) approaches. Where we say a language is functional and pure if a function is like a math function — it gets input and produces output and if you run it many times over — everything will be predictable. Pure functional programming languages make life super involved when building software, in exchange for very sound production software.</p> <h2 id="the-pragmatic-way">The Pragmatic Way</h2> <p>The good news is that there is a renaissance in the functional-programming world, in the shape of <em>sound typing</em> like Flow and programming languages like <a href="https://github.com/tc39">ES6</a> (Javascript) and <a href="https://reasonml.github.io">Reason</a> and libraries like <a href="https://underscorejs.org/">Underscore.js</a> which started an arms race that produced <a href="http://lodash.com">lodash</a>, <a href="http://toolz.readthedocs.io">toolz</a> and more, all of which are taking in a good chunk of the functional programming <em>mindset</em>, arguably because of extreme paradigm shift React, Redux, and the surrounding ecosystem made on the client side, and frameworks like Twitter’s <a href="https://twitter.github.io/finagle/">Finagle</a> (with thier seminal <a href="https://monkey.org/%7Emarius/funsrv.pdf">function over a server</a> essay) caused and is continuing to cause on the backend.</p> <p>Another chunk of good news is that we’re going to go over a pragmatic approach to functional programming that produces a good amount of the value but not all of it, and that in turn, it won’t take nearly as much time to learn. Some FP purists would say this is heresy, but we’ll continue any way.</p> <p>To keep our pragmatic approach, we’ll look at two tools I’ve built in Python that are open source and exhibit this kind of functional programming approach.</p> <p>Suppose you have a <code>~/.dotfiles</code> folder (as I do), which you carry over from machine to machine, and it includes your set up files, infrastructure, configuration, software lists to install and so on.</p> <p>How could we make it so that a machine can be provisioned automatically without resorting to big hammers like <a href="https://www.ansible.com/">Ansible</a> or <a href="https://www.chef.io/chef/">Chef</a>? The answer I found is that with an opinionated folder structure and a couple of tools ( <a href="https://github.com/jondot/dotlinker">dotlinker</a>, <a href="https://github.com/jondot/dotrunner">dotrunner</a>), this was completely possible, maintainable and reliable.</p> <p>We start with an opinionated structure for our <code>~/.dotfiles</code> folder. Here's part of mine as it stands right now:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">├── fish </span><span class="z-text z-plain">│ └── .config </span><span class="z-text z-plain">│ └── fish </span><span class="z-text z-plain">│ ├── config.fish.symlink </span><span class="z-text z-plain">│ └── fishfile.symlink </span><span class="z-text z-plain">├── node </span><span class="z-text z-plain">│ ├── install.sh </span><span class="z-text z-plain">│ ├── install.yml </span><span class="z-text z-plain">│ └── node.sh </span><span class="z-text z-plain">├── osx </span><span class="z-text z-plain">│ ├── install.sh </span><span class="z-text z-plain">│ └── install.yml </span><span class="z-text z-plain">├── python </span><span class="z-text z-plain">│ ├── install.sh </span><span class="z-text z-plain">│ └── install.yml </span></code></pre> <p>We want to accomplish two things:</p> <ol> <li>Go over all <code>**/*/.symlink</code> files with their hierarchy and set them up as symlinks. So transform <code>~/.dotfiles/fish/.config/fish/config.fish.symlink</code> to <code>~/.config/fish/config.fish</code> and symlink based on that.</li> <li>Build a DAG (Directed Acyclic Graph) of all of the install tasks (here: node, osx, python) by scanning, looking for, and parsing <code>install.yml</code> s, also assuming that they depend on one another so execute them in an efficient, dependent-free, way.</li> </ol> <p>If we started from scratch, here’s how we’d build these pieces of software, in a functional programming mindset.</p> <h3 id="identify-the-pipeline">Identify the Pipeline</h3> <p>Unless we’re building UI — and that also is arguable — which means we have to keep state and maintain a state machine at all times, we’re at a big advantage — we should be dealing with problems that <em>should</em> map to having a bunch of data go through a bunch of functions in a forward-only way.</p> <p>That means, that as you approach a problem — you can assume, and look for, the <em>pipeline</em> that should exist.</p> <p>In our case, for the <code>symlinker</code> part we can assume the following pipeline:</p> <ol> <li>Get to a position where we <em>list all files</em> under <code>.dotfiles</code></li> <li>Sort them out, filter unneeded</li> <li>Build a linking map of who’s source and where’s target</li> <li>Side effect time! validate that targets don’t exist</li> <li>Side effect time! execute all os-level symlink calls on source and target</li> </ol> <p>So in concept — a bunch of files moving through a bunch of functions. Check! That’s the north star that should guide our implementation.</p> <p>We say that our data <em>moves though</em> our functions, which is important. And if we look at a drawing, we imagine it that way:</p> <p><img src="https://rng0.io/blog/functional-programming-with-python-for-people-without-time/2.png" alt="" /></p> <p>Where the notation:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">[file: string] </span></code></pre> <p>Means an <em>array</em> of files, of type <em>string</em>, and [result: bool] would be an array of results of type <em>bool</em> and so on. Most important to note is how <em>types expand and contract</em>, or <em>map and reduce</em> along the pipeline:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">string </span><span class="z-text z-plain"> | \ </span><span class="z-text z-plain">string (filtered strings) </span><span class="z-text z-plain"> | \ </span><span class="z-text z-plain">FileInfo (filesystem &#39;stat&#39; turned every string into a FileInfo) </span><span class="z-text z-plain"> | \ </span><span class="z-text z-plain">(FileInfo, string) (we paired target and source, ready for action) </span><span class="z-text z-plain"> | / </span><span class="z-text z-plain"> | / </span><span class="z-text z-plain">bool (mindlessly linked source and target, resulted in true/false per pair) </span></code></pre> <p>Let’s break down how we can use <a href="https://github.com/pytoolz/toolz">toolz</a> to implement this:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">toolz</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">curried</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">pipe</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-support z-function z-builtin z-python">filter</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-support z-function z-builtin z-python">map</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">concat</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">complement</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">os</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">path</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">splitext</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">reject</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">pred</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">filter</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">complement</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">pred</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">pipe</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">io</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">ls</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">root</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">filter</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">io</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">isdir</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">reject</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">d</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">d</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">startswith</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">-<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">map</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">io</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">glob</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">concat</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">filter</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">f</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">splitext</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">f</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-constant z-numeric z-integer z-decimal z-python">1</span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span> <span class="z-keyword z-operator z-comparison z-python">==</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">.symlink<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">map</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">io</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">stat</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">map</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">info</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-sequence z-tuple z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">info</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">make_target_of</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">info</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-sequence z-end z-python">)</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">map</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">io</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">link</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-support z-type z-python">list</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <h3 id="build-a-pipe-with-pipe">Build a Pipe with <code>pipe</code></h3> <p>Start your flow with <code>pipe</code>. A pipe backed by <code>pipe</code> means "give me a list of things" ( <code>io.ls(root)</code>) to start out with, and then a list of functions to thread this data into.</p> <p><a href="https://theburningmonk.com/2011/09/fsharp-pipe-forward-and-pipe-backward/">Some</a> <a href="https://elixirschool.com/en/lessons/basics/pipe-operator/">languages</a> take this so seriously, that they have their own operator for pipe:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">|&gt; </span></code></pre> <h3 id="there-s-a-function-for-it">There’s a Function for It</h3> <p>Use functional operators (e.g. <code>map</code>, <code>filter</code> and so on). A <code>map(io.glob)</code> captures an otherwise painfully written for-each-loop, and it clearly states <em>what</em> you want do - and not <em>how</em> you want to do it - glob over all entries coming from <code>ls</code>. And it took the minimum number of characters to write that's still legible.</p> <h3 id="currying-matters">Currying Matters</h3> <p>Use the curried form of functional operators. This allows us to build concise execution units that are reusable: a <code>map(io.glob)</code> builds a new function that takes a list of things and it sits there waiting for those things to arrive. This means you can reuse it anywhere you want, pass it, cache it, and most importantly - compose it. The opposite is to <code>map(io.glob, &lt;list of things&gt;)</code> which forces you to give an input at the point of declaration, and is impossible to compose (for example, you can't use it in a pipe).</p> <p>More over, turning into point-free style (there is a more pure definition for ‘point-free’ which we’ll ignore) by making use of currying, we can have this:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">reject</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">d</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">d</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">startswith</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">-<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span>, </span></code></pre> <p>Become this:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">prefixed_by</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">pref</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">d</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">d</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">startswith</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">pref</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span></span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">reject</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">prefixed_by</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">-<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span>, </span></code></pre> <p>Essentially turning <code>startswith</code> which is a method that must be bound to an existing instance of string to an independent function called <code>prefixed_by</code>. This makes the <code>reject</code> clause clearer.</p> <p>Generally, we can keep doing this to produce a “cleaner” version of our pipeline, having to extract and define more of these kinds of generic utility functions:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">toolz</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">curried</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">pipe</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-support z-function z-builtin z-python">filter</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-support z-function z-builtin z-python">map</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">mapcat</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">complement</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">os</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">path</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">splitext</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">reject</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">pred</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">filter</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">complement</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">pred</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span></span> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">prefixed_by</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">pref</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">d</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">d</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">startswith</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">pref</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span></span></span> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">just_symlinks</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">f</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">splitext</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">f</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-constant z-numeric z-integer z-decimal z-python">1</span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span> <span class="z-keyword z-operator z-comparison z-python">==</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">.symlink<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">make_targets</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-variable z-parameter z-python">info</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-sequence z-tuple z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">info</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">make_target_of</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">info</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-sequence z-end z-python">)</span></span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">pipe</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">io</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">ls</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">root</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">filter</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">io</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">isdir</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">reject</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">prefixed_by</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">-<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">mapcat</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">io</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">glob</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">filter</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">just_symlinks</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">map</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">io</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">stat</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">map</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">make_targets</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">map</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">io</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">link</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-support z-type z-python">list</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p>I hope that with this I’ve inspired you to do more, and if you feel willing <a href="https://toolz.readthedocs.io/en/latest/api.html#functoolz">take a look at the toolz API</a>, and its <a href="https://toolz.readthedocs.io/en/latest/streaming-analytics.html">Streaming Analytics</a> use case.</p> <p>The next place to visit is the <a href="https://github.com/sfermigier/awesome-functional-python">awesome functional python</a> list. There you’ll find libraries such as toolz, pydash, funcy, and more, and functional languages with a strong resemblence to Python, or implemented with Python like <a href="http://docs.hylang.org/en/latest/">Hy</a> and <a href="http://coconut-lang.org/">Coconut</a>.</p> <h3 id="hexagonal-architecture-and-functional-programming">Hexagonal Architecture and Functional Programming</h3> <p>A pattern I’ve started to use around ten years ago, when I was strongly into Domain Driven Design (while doing enterprise C# and Java), is <a href="https://staging.cockburn.us/hexagonal-architecture/">hexagonal architecture</a> — also known as “Onion Architecture”, and later known as <a href="https://8thlight.com/blog/uncle-bob/2012/08/13/the-clean-architecture.html">“Clean Architecture”</a>. The reason all these popped up at that tight corridor of time (2008–10) is “Web scale”; the growing demand for scalable <em>and</em> maintainable softare, and the massive shifts in how we saw databases and persistence layers to name a few.</p> <p>While all these architecture concepts have had slightly different lines of thought, their theme is common: a core business logic, surrounded by plugins or adapters, or more commonly: interfaces.</p> <p>I still adopt the same approach in functional programming. Specifically, when doing I/O or side-effects, I consolidate those into individual modules to be plugged onto the core.</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">pipe</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">io</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">ls</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">root</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">filter</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">io</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">isdir</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">reject</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">prefixed_by</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">-<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">mapcat</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">io</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">glob</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">filter</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">just_symlinks</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">map</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">io</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">stat</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">map</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">make_targets</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">map</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">io</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">link</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-support z-type z-python">list</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p>In the code listings above we saw one such module as the <code>io</code> module. It's responsible for all of the I/O access in that codebase and even though it might sound like being a control-freak over I/O is tiresome - it's worth it. It's worth it in cleanliness, in testing, and in helping drive a better design overall.</p> <p>Some pure languages such as Haskell go to great length to model I/O (or more generally: side effects) in a perfect way (you might have heard the term “Monad” some time), but I’ve tactically chosen to sidestep that topic. What’s more, is that not all functional programming languages choose to be pure, a notable one which I tend to like is <a href="https://ocaml.org/">OCaml</a> and its more hipster, Javascript backed, sibling <a href="https://reasonml.github.io">ReasonML</a> from Facebook.</p> <h3 id="exceptions-and-error-handling">Exceptions and Error Handling</h3> <p>We left out a topic that needs a good chunk of theory and mind-massaging before acknowledging it’s a better way to do things: error handling in functional programming. Exceptions and similar errors shatter the stack, and break the program flow — how does that sit with the concept of data always flowing forward onto functions in functional programming?</p> <p>The answer is that <em>there are no exceptions</em>. There’s errors, and more abstractly there’s probably two kinds of return types from any function: success and failure.</p> <p>That means that in this ideal model, every function in the functional programming world receives and outputs a result type that can shape as two different types: success and failure. Let’s model this in the following fictional statically typed functional language:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">fun foo() -&gt; Result&lt;Success(x), Failure(y)&gt; </span><span class="z-text z-plain"> </span><span class="z-text z-plain">main(){ </span><span class="z-text z-plain"> match(foo()){ </span><span class="z-text z-plain"> Success(x): println(x) </span><span class="z-text z-plain"> Failure(y): error(y) </span><span class="z-text z-plain"> } </span><span class="z-text z-plain">} </span></code></pre> <p>The <code>-&gt; Result</code> notation means "returns a type of Result" and the <code>Result&lt;Success(x),Failure(y)&gt;</code> notation means <code>Result</code> can generically represent <code>Success</code> or <code>Failure</code> each of which can contain a value <code>x</code> or <code>y</code> which we can extract later with <code>match</code>.</p> <p><code>match</code> is a common operator in typed functional languages (also in untyped ones but much more effective in typed languages) that help us perform introspection on a type, branch according to the specific type, and destructure it - extract its content and bind its content to a normal value variable - all at once(!).</p> <p>At this moment <code>foo</code> is receiving no argument, but if it would - according to the theory we're developing here - it would get the same "carrier" type: Result.</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">fun foo(Result) -&gt; Result&lt;Success(x), Failure(y)&gt; </span><span class="z-text z-plain"> </span><span class="z-text z-plain">main(){ </span><span class="z-text z-plain"> some_result = ... </span><span class="z-text z-plain"> match(foo(some_result)){ </span><span class="z-text z-plain"> Success(x): println(x) </span><span class="z-text z-plain"> Failure(y): error(y) </span><span class="z-text z-plain"> } </span><span class="z-text z-plain">} </span></code></pre> <p>Now comes the fun part. What if <code>some_result</code> is actually an error result - or in other words, in an error state? Surely, we don't want <code>foo</code> to do any redundant work. In this case want it to <em>short-circuit</em> and return the result it's passed as input (because it's an error).</p> <p>Threading the error forward in such a way, is the start of replacing the traditional error handling mechanism we might have thought was the only one in the world — exceptions.</p> <p>Actually you already might have implemented such a thing — all I need in order to make you remember it is throw a curve ball at you. Here’s a question — if you had a list of background jobs to spin off, each job to be performed by a different thread — did you use exceptions to communicate errors? What did you do?</p> <p>Hopefully, you’ve made each thread return a result — a <code>success</code> or an <code>error</code>. Finally arriving at a list of thread results:</p> <pre data-lang="json" class="language-json z-code"><code class="language-json" data-lang="json"><span class="z-source z-json"><span class="z-meta z-sequence z-json"><span class="z-punctuation z-section z-sequence z-begin z-json">[</span><span class="z-constant z-language z-json">true</span><span class="z-punctuation z-separator z-sequence z-json">,</span> <span class="z-constant z-language z-json">true</span><span class="z-punctuation z-separator z-sequence z-json">,</span> <span class="z-constant z-language z-json">false</span><span class="z-punctuation z-separator z-sequence z-json">,</span> <span class="z-constant z-language z-json">false</span><span class="z-punctuation z-separator z-sequence z-json">,</span> <span class="z-constant z-language z-json">true</span><span class="z-punctuation z-section z-sequence z-end z-json">]</span></span> </span></code></pre> <p>In this case, if you followed that kind of solution — you implicitly had the notion of a <code>Result&lt;true,false&gt;</code>. If this solution was good for this specific scenario, why not apply it to <em>all</em> scenarios. And more importantly, why have <em>two</em> ways to solve errors? (both exceptions and errors as result types). That's part of the <a href="https://sidburn.github.io/blog/2016/03/25/exceptions-are-evil">everlasting</a> discussions about why exceptions are a bad smell; there's no clear winner but now you get to have a good footing in this discussion.</p> <p>As a side note: in our code listings so far, we didn’t explicitely model <em>any</em> of that error handling mechanism. That is because for dynamic languages this kind of construct is easier to implement (just a dict with a field indicating success or failure and a payload field as a result), but also very noisy to code (no pattern matching means if’s and tiresome manual left-right-coding). So we kept it out for now.</p> <p><em>Special thanks for reviewing:</em> <a href="https://twitter.com/magthe"><em>Magnus Therning</em></a> <em>, Ofer Horowitz</em></p> Shipping Electron Apps to Mac App Store with Electron Builder Tue, 15 May 2018 00:00:00 +0000 Unknown https://rng0.io/blog/shipping-electron-apps-to-mac-app-store-with-electron-builder/ https://rng0.io/blog/shipping-electron-apps-to-mac-app-store-with-electron-builder/ <p><img src="https://rng0.io/blog/shipping-electron-apps-to-mac-app-store-with-electron-builder/1.png" alt="" /></p> <p>Shipping Electron apps have changed with time in plenty of ways. The latest that works pretty neatly is <a href="https://github.com/electron-userland/electron-builder">Electron Builder</a>, it’s also worth watching the <a href="https://github.com/electron-userland">Electron Userland</a> space for more developments.</p> <p>One small warning for anyone dealing with this, is that history shows that the workflow around packaging Electron is happy to break. Building <em>and</em> packaging an Electron app for <em>multiple</em> distribution methods is still no small feat.</p> <p>Here’s just a <em>few</em> things that can change with time with a real-world Electron app — each can make your build fail.</p> <ul> <li>Electron version.</li> <li>Electron builder version.</li> <li>Webpack (you <em>do</em> want an independent bundle).</li> <li>Babel (you <em>do</em> want a modern Javascript)</li> <li>Native deps and how to package and sign those.</li> <li>Xcode tools, macOS changes and electron tooling changes upstream (read: later) to that.</li> <li>Certificate voodoo. Expirey, invalidation, management of certificates (which is a pain for iOS too, mitigated with Fastlane <a href="https://docs.fastlane.tools/actions/match/">match</a>)</li> </ul> <p>With that in mind, if you follow the below instructions, you’ll have a rather smooth and uneventful packaging experience.</p> <h2 id="folder-structure">Folder structure</h2> <p>Here’s how it looks like without Javascript specific stuff. Only packaging related files and directories are here.</p> <p>We mark each of the topics with a number and cover them later below.</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">src/ </span><span class="z-text z-plain">build/ </span><span class="z-text z-plain"> all-certs.p12 &lt;--- (1) all certs exported </span><span class="z-text z-plain"> entitlements.mac.plist &lt;--- boilerplate, nothing important </span><span class="z-text z-plain"> entitlements.mas.plist &lt;--- (3) team and dev ids </span><span class="z-text z-plain"> electronMac{.original}.js &lt;--- (5) fixes bug with builder </span><span class="z-text z-plain">electron-builder.yaml &lt;--- (4) contains build information </span><span class="z-text z-plain">embedded.provisionprofile &lt;--- (2) provisioning profile </span></code></pre> <h2 id="build-process-and-terminology">Build Process and Terminology</h2> <p>Electron builder <a href="https://www.electron.build/tutorials/two-package-structure">documentation</a> assumes (maybe rightfully?) some terminology, I’m guessing because Electron tooling has come a way now and this is the Nth incarnation of packaging tools and it relies on your having some experience with older tools.</p> <p>Let’s explain a few of the relevant terminology now.</p> <ul> <li>Build process. Electron-builder will look at <code>targets</code> and just the one (1) certificate bundle and build all variants at once, placing results in 'release'.</li> <li><em>mas</em>: “Mac App Store” build. Submitting needs a ‘pkg’ package.</li> <li><em>mac</em>: “Mac stand-alone” build, Apple doesn’t supervise and participate in distributing this, but macOS <em>does</em> verify product signatures so that people don’t run things that come from no-where (a verifying check which a power-user can disable through security settings on a macOS). Usually distributed with a ‘dmg’ package.</li> <li><em>pkg, dmg</em>: two packaging formats for the app; each is needed base on the variant packaging that we opt for (mac, mas).</li> </ul> <h2 id="1-certificates">1. Certificates</h2> <p>Electron builder’s <a href="https://www.electron.build/code-signing">code-signing</a> section doesn’t tell you much about what you need to prepare <em>before</em> configuring it.</p> <p>Use Xcode to generate your mac app certificates:</p> <ol> <li>Open Xcode</li> <li>Preferences (Cmd+,) -&gt; Accounts Tab</li> <li>Manage certificates (log in if you never did)</li> </ol> <p>Use the small ‘+’ dropdown at the button to generate the following:</p> <ol> <li>Mac App Distribution</li> <li>Mac Installer Distribution</li> <li>Developer ID Application</li> <li>Developer ID Installer</li> <li>Mac Development (not a must)</li> </ol> <p>The <em>most</em> important step is to export these into a single p12 file. You might have been used to exporting individually when working with iOS apps.</p> <p>Launch Keychain, and go to to ‘My Certificates’. Find these certificates and highlight <em>all</em> of them (Cmd+click), right click and export as p12.</p> <p>Place this p12 file as <code>build/all-certs.p12</code>.</p> <p>Super important to realize that if you have additional binaries, whether a <code>node.js</code> native dependency or just an executable that you include in your resources, that you want your app to use (let's say <code>ffmpeg</code> or such), they too need to be signed and will be signed by Electron Builder, which is a nice surprise.</p> <p>If any of this kind of signing is broken or the application doesn’t run properly, be ready to look there.</p> <h2 id="2-provisioning-profile">2. Provisioning Profile</h2> <p>Log into your apple developer account ( <em>developer.apple.com</em>) and go to “Certificates, IDs &amp; Profiles”. Be sure to move to the “macOS” dropdown or else you’ll be manipulating iOS related assets.</p> <p>Go to ‘Provisioning Profiles’ and create one for <em>distribution</em> named <code>embedded</code>. Download that into <code>embedded.provisionprofile</code> at the root of your project (from where ever you'll be running <code>electron-builder</code>). Place it only at the root and don't try to be smart about it in favor of a cleaner project structure (as I did), the story is that multiple tools are involved with the Electron Builder packaging workflow, one of them relies on this file existing at the root.</p> <h2 id="3-entitlements">3. Entitlements</h2> <p>On an app that doesn’t require special external services from Apple, these files will be almost empty.</p> <p><em>entitlements.mac.plist</em></p> <pre data-lang="xml" class="language-xml z-code"><code class="language-xml" data-lang="xml"><span class="z-text z-xml"><span class="z-meta z-tag z-preprocessor z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;?</span><span class="z-entity z-name z-tag z-xml">xml</span> <span class="z-entity z-other z-attribute-name z-localname z-xml">version</span><span class="z-punctuation z-separator z-key-value z-xml">=</span><span class="z-string z-quoted z-double z-xml"><span class="z-punctuation z-definition z-string z-begin z-xml">&quot;</span>1.0<span class="z-punctuation z-definition z-string z-end z-xml">&quot;</span></span> <span class="z-entity z-other z-attribute-name z-localname z-xml">encoding</span><span class="z-punctuation z-separator z-key-value z-xml">=</span><span class="z-string z-quoted z-double z-xml"><span class="z-punctuation z-definition z-string z-begin z-xml">&quot;</span>UTF-8<span class="z-punctuation z-definition z-string z-end z-xml">&quot;</span></span><span class="z-punctuation z-definition z-tag z-end z-xml">?&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-sgml z-doctype z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;!</span><span class="z-keyword z-declaration z-doctype z-xml">DOCTYPE</span> <span class="z-variable z-other z-documentroot z-localname z-xml">plist</span> <span class="z-storage z-type z-external-content z-xml">PUBLIC</span> <span class="z-string z-quoted z-double z-xml"><span class="z-punctuation z-definition z-string z-begin z-xml">&quot;</span>-//Apple//DTD PLIST 1.0//EN<span class="z-punctuation z-definition z-string z-end z-xml">&quot;</span></span> <span class="z-string z-quoted z-double z-xml"><span class="z-punctuation z-definition z-string z-begin z-xml">&quot;</span>http://www.apple.com/DTDs/PropertyList-1.0.dtd<span class="z-punctuation z-definition z-string z-end z-xml">&quot;</span></span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;</span><span class="z-entity z-name z-tag z-localname z-xml">plist</span> <span class="z-entity z-other z-attribute-name z-localname z-xml">version</span><span class="z-punctuation z-separator z-key-value z-xml">=</span><span class="z-string z-quoted z-double z-xml"><span class="z-punctuation z-definition z-string z-begin z-xml">&quot;</span>1.0<span class="z-punctuation z-definition z-string z-end z-xml">&quot;</span></span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;</span><span class="z-entity z-name z-tag z-localname z-xml">dict</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;</span><span class="z-entity z-name z-tag z-localname z-xml">key</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span>com.apple.security.files.user-selected.read-only<span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;/</span><span class="z-entity z-name z-tag z-localname z-xml">key</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;</span><span class="z-entity z-name z-tag z-localname z-xml">true</span><span class="z-punctuation z-definition z-tag z-end z-xml">/&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;</span><span class="z-entity z-name z-tag z-localname z-xml">key</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span>com.apple.security.files.user-selected.read-write<span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;/</span><span class="z-entity z-name z-tag z-localname z-xml">key</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;</span><span class="z-entity z-name z-tag z-localname z-xml">true</span><span class="z-punctuation z-definition z-tag z-end z-xml">/&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;/</span><span class="z-entity z-name z-tag z-localname z-xml">dict</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;/</span><span class="z-entity z-name z-tag z-localname z-xml">plist</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span></code></pre> <p><em>entitlements.mas.plist</em></p> <pre data-lang="xml" class="language-xml z-code"><code class="language-xml" data-lang="xml"><span class="z-text z-xml"><span class="z-meta z-tag z-preprocessor z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;?</span><span class="z-entity z-name z-tag z-xml">xml</span> <span class="z-entity z-other z-attribute-name z-localname z-xml">version</span><span class="z-punctuation z-separator z-key-value z-xml">=</span><span class="z-string z-quoted z-double z-xml"><span class="z-punctuation z-definition z-string z-begin z-xml">&quot;</span>1.0<span class="z-punctuation z-definition z-string z-end z-xml">&quot;</span></span> <span class="z-entity z-other z-attribute-name z-localname z-xml">encoding</span><span class="z-punctuation z-separator z-key-value z-xml">=</span><span class="z-string z-quoted z-double z-xml"><span class="z-punctuation z-definition z-string z-begin z-xml">&quot;</span>UTF-8<span class="z-punctuation z-definition z-string z-end z-xml">&quot;</span></span><span class="z-punctuation z-definition z-tag z-end z-xml">?&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-sgml z-doctype z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;!</span><span class="z-keyword z-declaration z-doctype z-xml">DOCTYPE</span> <span class="z-variable z-other z-documentroot z-localname z-xml">plist</span> <span class="z-storage z-type z-external-content z-xml">PUBLIC</span> <span class="z-string z-quoted z-double z-xml"><span class="z-punctuation z-definition z-string z-begin z-xml">&quot;</span>-//Apple//DTD PLIST 1.0//EN<span class="z-punctuation z-definition z-string z-end z-xml">&quot;</span></span> <span class="z-string z-quoted z-double z-xml"><span class="z-punctuation z-definition z-string z-begin z-xml">&quot;</span>http://www.apple.com/DTDs/PropertyList-1.0.dtd<span class="z-punctuation z-definition z-string z-end z-xml">&quot;</span></span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;</span><span class="z-entity z-name z-tag z-localname z-xml">plist</span> <span class="z-entity z-other z-attribute-name z-localname z-xml">version</span><span class="z-punctuation z-separator z-key-value z-xml">=</span><span class="z-string z-quoted z-double z-xml"><span class="z-punctuation z-definition z-string z-begin z-xml">&quot;</span>1.0<span class="z-punctuation z-definition z-string z-end z-xml">&quot;</span></span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;</span><span class="z-entity z-name z-tag z-localname z-xml">dict</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;</span><span class="z-entity z-name z-tag z-localname z-xml">key</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span>com.apple.security.app-sandbox<span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;/</span><span class="z-entity z-name z-tag z-localname z-xml">key</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;</span><span class="z-entity z-name z-tag z-localname z-xml">true</span><span class="z-punctuation z-definition z-tag z-end z-xml">/&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;</span><span class="z-entity z-name z-tag z-localname z-xml">key</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span>com.apple.security.application-groups<span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;/</span><span class="z-entity z-name z-tag z-localname z-xml">key</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;</span><span class="z-entity z-name z-tag z-localname z-xml">string</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span>XXXXX.your.bundle.id<span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;/</span><span class="z-entity z-name z-tag z-localname z-xml">string</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;</span><span class="z-entity z-name z-tag z-localname z-xml">key</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span>com.apple.security.files.user-selected.read-only<span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;/</span><span class="z-entity z-name z-tag z-localname z-xml">key</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;</span><span class="z-entity z-name z-tag z-localname z-xml">true</span><span class="z-punctuation z-definition z-tag z-end z-xml">/&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;</span><span class="z-entity z-name z-tag z-localname z-xml">key</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span>com.apple.security.files.user-selected.read-write<span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;/</span><span class="z-entity z-name z-tag z-localname z-xml">key</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;</span><span class="z-entity z-name z-tag z-localname z-xml">true</span><span class="z-punctuation z-definition z-tag z-end z-xml">/&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;/</span><span class="z-entity z-name z-tag z-localname z-xml">dict</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span><span class="z-text z-xml"><span class="z-meta z-tag z-xml"><span class="z-punctuation z-definition z-tag z-begin z-xml">&lt;/</span><span class="z-entity z-name z-tag z-localname z-xml">plist</span><span class="z-punctuation z-definition z-tag z-end z-xml">&gt;</span></span> </span></code></pre> <p>In the <code>*.mas.*</code> variant fill in the value denoted by <code>XXX...</code> with your bundle ID and prefix. You can take these values from your developer account, where you have allocated a bundle ID for your app.</p> <h2 id="4-electron-builder-configuration">4. Electron Builder Configuration</h2> <p>It is preferred to keep in a separate YAML file rather than in your <code>package.json</code> file as an embedded block of configuration because this will let your work with a single <code>package.json</code> file more easily both for build and development and skip the <a href="https://www.electron.build/tutorials/two-package-structure">two-package project layout</a> which is arguably less clean.</p> <p>Here’s an <code>electron-builder.yaml</code> file that builds for all platforms and ways of distribution, including Mac App Store.</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">directories</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">output</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">release</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">appId</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">your.bundle.id</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">asarUnpack</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-quoted z-single z-yaml"><span class="z-punctuation z-definition z-string z-begin z-yaml">&#39;</span>**/resources/some-native-binary<span class="z-punctuation z-definition z-string z-end z-yaml">&#39;</span></span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">productName</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">MyAwesomeApp</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">files</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">dist/</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">from</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">dist/</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">to</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-quoted z-double z-yaml"><span class="z-punctuation z-definition z-string z-begin z-yaml">&quot;</span>.<span class="z-punctuation z-definition z-string z-end z-yaml">&quot;</span></span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">filter</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">index.js</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">resources/</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">index.html</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">embedded.provisionprofile</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">mac</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">category</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">public.app-category.productivity</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">entitlements</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">build/entitlements.mac.plist</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">icon</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">resources/app.icns</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">target</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">pkg</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">dmg</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">zip</span> </span><span class="z-source z-yaml"> <span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-string z-unquoted z-plain z-out z-yaml">mas</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">mas</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">type</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">distribution</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">category</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">public.app-category.productivity</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">entitlements</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">build/entitlements.mas.plist</span> </span><span class="z-source z-yaml"> <span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">icon</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">resources/app.icns</span> </span></code></pre> <h2 id="5-packaging">5. Packaging</h2> <p>We need to specify the environment variable (as <a href="https://www.electron.build/code-signing">documented</a>) <code>CSC_LINK</code> to be tied to our bundle of certificates.</p> <p>In addition, due to <a href="https://github.com/electron-userland/electron-builder/issues/2719">this recent issue</a> your app will build but won’t validate on the Mac App Store. To resolve it, you can use the patch provided there before building.</p> <p>Something like this will work around the validation issue:</p> <pre data-lang="json" class="language-json z-code"><code class="language-json" data-lang="json"><span class="z-source z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>package:standalone<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span>: </span><span class="z-source z-json"> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>cp build/electronMac.original.js node_modules/electron-builder-lib/out/electron/electronMac.js &amp;&amp; CSC_LINK=./build/all-certs.p12 electron-builder<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span>, </span><span class="z-source z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>package:app-store<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span>: </span><span class="z-source z-json"> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>cp build/electronMac.js node_modules/electron-builder-lib/out/electron/electronMac.js &amp;&amp; CSC_LINK=./build/all-certs.p12 electron-builder<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span>, </span></code></pre> <p>Each build process will build with errors for the opposing build variant (either “mac” or “mas”), because of this patch but it will still build the variant it was invoked for properly.</p> <p>Finally use Mac Application Loader to upload the freshly baked <em>mas/*.pkg</em> build result.</p> <h2 id="summary">Summary</h2> <p>If you follow these instructions, everything should build smoothly (sans the issue above, which I imagine will resolve with time).</p> <p>Should you bump into an issue, remember the list of things that can make your build break and check those in order:</p> <ul> <li>Electron version.</li> <li>Electron builder version.</li> <li>Webpack (you <em>do</em> want an independent bundle).</li> <li>Babel (you <em>do</em> want a modern Javascript)</li> <li>Native deps and how to package and sign those.</li> <li>Xcode tools, macOS changes and electron tooling changes upstream (read: later) to that.</li> <li>Certificate voodoo. Expirey, invalidation, management of certificates (which is a pain for iOS too, mitigated with Fastlane <a href="https://docs.fastlane.tools/actions/match/">match</a>)</li> </ul> <p>Even though it feels like a lot of work, doing all this to ship a macOS app <em>outside</em> Xcode without tools like <a href="https://fastlane.tools/">Fastlane</a> is simply amazing.</p> Supercharge your CreateReactApp workflow with Hygen Tue, 27 Feb 2018 00:00:00 +0000 Unknown https://rng0.io/blog/supercharge-your-createreactapp-workflow-with-hygen/ https://rng0.io/blog/supercharge-your-createreactapp-workflow-with-hygen/ <p><img src="https://rng0.io/blog/supercharge-your-createreactapp-workflow-with-hygen/1.gif" alt="" /></p> <p><a href="https://github.com/facebook/create-react-app"><em>CreateReactApp</em></a> is a starter project for React that has zero configuration and zero friction bundle of everything that’s great about React, Webpack, and modern Javascript. It’s the new standard for developer experience and ergonomics.</p> <p>If you’re unfamiliar with it, <a href="https://github.com/facebook/create-react-app">please do give it a go</a>.</p> <p>CreateReactApp is not just a developer convenience — it also allows you to go from zero to production builds in seconds. When you generate a new project with it, you’re given a single page app, and a main <code>App</code> page, with an accompanying test, and it builds and deploys perfectly.</p> <p>From that point on, you can mock up an idea and be done with it or you can install your go-to libraries, and consider your next step with an appropriate application architecture and state management solution.</p> <h2 id="react-components">React Components</h2> <p>No matter what direction you take from a fresh CreateReactApp, you’re probably going to want to build a pile of React components.</p> <p>A React component can be minimal like this:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts"><span class="z-entity z-name z-function z-ts">Avatar</span></span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-meta z-parameter z-object-binding-pattern z-ts"><span class="z-punctuation z-definition z-binding-pattern z-object z-ts">{</span> <span class="z-variable z-parameter z-ts">name</span> <span class="z-punctuation z-definition z-binding-pattern z-object z-ts">}</span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span><span class="z-cast z-expr z-ts"> <span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">div</span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span></span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">name</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">div</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span></span> </span></code></pre> <p>But really you probably will find yourself alternating between:</p> <ul> <li>A functional component (like the one above).</li> <li>A <code>PureComponent</code> which you'd use when you know you don't have state.</li> <li>A stateful component when you know you have a component state.</li> </ul> <p>You’d probably also want to create a test, and a <a href="http://storybook.js.org">storybook</a> file; a storybook for every component has become as essential as making sure there’s test for every piece of code.</p> <p>And if you think of those as peripheral assets that follow each component that make your output as engineer a high quality one, that’s probably just the start.</p> <h2 id="hygen">Hygen</h2> <p><a href="http://www.hygen.io">Hygen</a> is a fresh take on code generators that’s super suitable for React and Redux projects. It lets you build new generators in a breeze and use them immediately with zero configuration and zero friction, while keeping your generators project-local.</p> <p>The easiest way to install Hygen on a Mac is with Homebrew:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> brew tap jondot/tap</span> <span class="z-keyword z-operator z-logical z-and z-shell">&amp;&amp;</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">brew</span></span><span class="z-meta z-function-call z-arguments z-shell"> install hygen</span> </span></code></pre> <p>If you’re on Linux or Windows (or dislike Homebrew), you can install globally with Yarn:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn global add hygen</span> </span></code></pre> <p>Or locally to run via <code>yarn run</code>:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn add<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>dev</span> hygen</span> </span></code></pre> <p>But really, a global Hygen should work fine, and should save resources and feel faster (speed is a core Hygen feature!).</p> <h2 id="your-new-workflow">Your New Workflow</h2> <p>For convenience, Hygen comes with a package manager called <code>hygen-add</code>, where you can install pre-made generator <em>packages</em>.</p> <p>Packages can be a great starting point that assume everyone share the same set of problems, but you should definitely feel free to give each project its own take of its generators because most probably each project is unique.</p> <p>The new idea Hygen introduces is that — with all these extra files and extra glue and extra wirings needed for a modern React or Redux app, why manually add those? Let Hygen generate and wire and connect each component you make, and if you’re using Redux, it can do even more.</p> <p>Let’s start by setting up CreateReactApp and Hygen.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> brew tap jondot/hygen</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> brew install hygen</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn global add create-react-app hygen-add storybook/@cli</span> </span></code></pre> <p>Note that I’ve taken the liberty to include <code>hygen-add</code> and <code>storybook</code> up there, so that we can have a quicker experience.</p> <p>Let’s make a sample project:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> create-react-app myapp</span> <span class="z-keyword z-operator z-logical z-and z-shell">&amp;&amp;</span> <span class="z-meta z-function-call z-shell"><span class="z-support z-function z-cd z-shell">cd</span></span><span class="z-meta z-function-call z-arguments z-shell"> myapp</span> </span></code></pre> <p>Let’s set up <a href="https://storybook.js.org">Storybook</a> and install a convenient test renderer for React:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> getstorybook</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn add<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>dev</span> react-test-renderer</span> </span></code></pre> <p>On a separate terminal run your storybook server and open the resulting URL in a browser.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn storybook</span> </span></code></pre> <p>And if you like, run the actual React App (again on a different terminal):</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn start</span> </span></code></pre> <p>Instead of using many terminals, you can always use a tool like <a href="https://www.npmjs.com/package/concurrently">concurrently</a>, <a href="https://www.npmjs.com/package/foreman">Foreman</a> or <code>tmux</code>; I'll leave that to you.</p> <p>And now, let’s get our CreateReactApp package for Hygen:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> hygen-add cra</span> </span></code></pre> <p>That’s it. Let’s see what generators we have available:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> hygen</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Available</span></span><span class="z-meta z-function-call z-arguments z-shell"> actions:</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">component:</span></span><span class="z-meta z-function-call z-arguments z-shell"> help, new</span> </span></code></pre> <p>And let’s see how the <code>component</code> generator looks like:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> hygen component help</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">hygen</span></span><span class="z-meta z-function-call z-arguments z-shell"> component new<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>name</span> NAME <span class="z-keyword z-control z-regexp z-set z-begin z-shell">[</span>-<span class="z-keyword z-operator z-word z-shell">-</span>stateful<span class="z-keyword z-control z-regexp z-set z-end z-shell">]</span> <span class="z-keyword z-control z-regexp z-set z-begin z-shell">[</span>-<span class="z-keyword z-operator z-word z-shell">-</span>functional<span class="z-keyword z-control z-regexp z-set z-end z-shell">]</span></span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Generates</span></span><span class="z-meta z-function-call z-arguments z-shell"> a React component, a storybook, and a test.</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">NAME</span></span><span class="z-meta z-function-call z-arguments z-shell"> The component name in kebab-case (required</span><span class="z-meta z-function-call z-shell"></span>)<span class="z-meta z-function-call z-shell"><span class="z-support z-function z-dot z-shell">.</span></span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">--stateful</span></span><span class="z-meta z-function-call z-arguments z-shell"> Generate a stateful component (optional</span><span class="z-meta z-function-call z-shell"></span>)<span class="z-meta z-function-call z-shell"><span class="z-support z-function z-dot z-shell">.</span></span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">--functional</span></span><span class="z-meta z-function-call z-arguments z-shell"> Generate a functional component (optional</span><span class="z-meta z-function-call z-shell"></span>)<span class="z-meta z-function-call z-shell"><span class="z-support z-function z-dot z-shell">.</span></span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">If</span></span><span class="z-meta z-function-call z-arguments z-shell"> no flags given, will generate a PureComponent.</span> </span></code></pre> <p>From this point on, to add a new React component to your CreateReactApp based project is just:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> hygen component new<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>name</span> avatar</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Loaded</span></span><span class="z-meta z-function-call z-arguments z-shell"> templates: _templates</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">added:</span></span><span class="z-meta z-function-call z-arguments z-shell"> src/components/avatar/avatar.story.js</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">added:</span></span><span class="z-meta z-function-call z-arguments z-shell"> src/components/avatar/avatar.test.js</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">added:</span></span><span class="z-meta z-function-call z-arguments z-shell"> src/components/avatar/index.js</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">inject:</span></span><span class="z-meta z-function-call z-arguments z-shell"> src/stories/index.js</span> </span></code></pre> <p>And you get a full stack of content for your component; a test, a storybook, and wiring around it.</p> <p>You’ll also notice Storybook slurps up the new component and automatically updates, I promise this kind of fluidity will really charge up your <em>zone</em>. Check it out:</p> <p><img src="https://rng0.io/blog/supercharge-your-createreactapp-workflow-with-hygen/2.gif" alt="" /></p> <p>Should you want to add more aspects to your component, such as a full suite of Redux actions, reducer and so on, you can add them directly to your generator templates folder:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> tree _templates</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">_templates</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">└──</span></span><span class="z-meta z-function-call z-arguments z-shell"> component</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├──</span></span><span class="z-meta z-function-call z-arguments z-shell"> help</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">│</span></span><span class="z-meta z-function-call z-arguments z-shell"> └── index.txt.t</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">└──</span></span><span class="z-meta z-function-call z-arguments z-shell"> new</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├──</span></span><span class="z-meta z-function-call z-arguments z-shell"> comp.story.js.t</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├──</span></span><span class="z-meta z-function-call z-arguments z-shell"> comp.test.js.t</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">├──</span></span><span class="z-meta z-function-call z-arguments z-shell"> index.js.t</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">└──</span></span><span class="z-meta z-function-call z-arguments z-shell"> inject_story.t</span> </span></code></pre> <p>Once you add a new template (say, <code>actions.js.t</code>) its automatically included in the generated code.</p> <p>For more about this, check out the official Hygen docs about <a href="http://www.hygen.io/redux">Redux</a> and in general, it’s super useful to understand the <a href="http://www.hygen.io/quick-start">primary concepts</a> around Hygen.</p> Build Your Own ICO in 5 Minutes Mon, 12 Feb 2018 00:00:00 +0000 Unknown https://rng0.io/blog/build-your-own-ico-in-5-minutes/ https://rng0.io/blog/build-your-own-ico-in-5-minutes/ <p>Before we start, you’ll need <a href="https://nodejs.org/">Node.js</a> and <a href="https://yarnpkg.com">Yarn</a> installed.</p> <p>We’ll use <a href="https://github.com/jondot/hygen">Hygen</a> to drive the creation of our contracts. Hygen is a new productivity tool that lets you generate code in a fun way, for any purpose and any platform.</p> <p>Let’s install Hygen globally:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn global add hygen</span> </span></code></pre> <p>Clone our base repository and <code>cd</code> into it. We already have Hygen generators in it.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> git clone https://github.com/jondot/5-min-ico</span><span class="z-keyword z-operator z-logical z-continue z-shell">;</span> <span class="z-meta z-function-call z-shell"><span class="z-support z-function z-cd z-shell">cd</span></span><span class="z-meta z-function-call z-arguments z-shell"> 5-min-ico</span> </span></code></pre> <p>Run hygen with the <code>ico</code> generator, and answer the questions about your ICO:</p> <p><img src="https://rng0.io/blog/build-your-own-ico-in-5-minutes/1.png" alt="" /></p> <p>That’s it. You can verify that your contracts build:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn build</span> </span></code></pre> <p>And finally, you can deploy to your chosen testnet (read below if you don’t have one set up or don’t know what that is):</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn deploy</span> </span></code></pre> <h2 id="using-a-testnet">Using a Testnet</h2> <p>If you don’t want to deploy to a real blockchain, and want smooth sailing, you can just use <a href="https://github.com/trufflesuite/ganache">Ganache</a>, which is the most recent evolution of a local test tool (you may have heard about <em>testrpc</em> previously) to help simulate and develop on the blockchain. It’s just a desktop app you need to install.</p> <p>Using Ganache, you can deploy with zero configuration or set up (but do check its settings, you might have to change the port it uses to 8545).If you want to get your hands dirty with a real blockchain (still a testnet), keep reading.</p> <p>You need to deploy to an Ethereum network. Using <a href="https://github.com/paritytech/parity">Parity</a> we’ll work against the Kovan testnet (one of many test networks that you can use), which is fast and stable.</p> <p>See how to install parity for your OS <a href="https://github.com/paritytech/parity">here</a> and after installing it, we’ll run it with the Kovan testnet.</p> <p>If you’re on macOS and use Homebrew, I’ll save you the reading:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> brew tap paritytech/paritytech</span> <span class="z-keyword z-operator z-logical z-and z-shell">&amp;&amp;</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">brew</span></span><span class="z-meta z-function-call z-arguments z-shell"> install parity</span> </span></code></pre> <p>Now, run Parity on a separate terminal window:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> parity ui<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>chain</span> kovan</span> </span></code></pre> <p>Once started, it may take some time for it to sync with other nodes, so be patient. You’ll be able to see on the UI what’s the health of your node.</p> <p>Meanwhile, Parity UI will open on your browser, fiddle with it a bit, but primarily, create a new account. Read <a href="https://github.com/paritytech/parity/wiki/Accounts,-Wallets,-Vaults">here</a> for more information about how to do that. The deploy script will use the first account it finds, and will use funds from it on order to fuel its deploy transactions.</p> <p>We’re almost ready to deploy. Deploying to Kovan requires that you get some ‘fake’ ETH into your account.</p> <p>You need to go <a href="https://github.com/kovan-testnet/faucet">here</a> in order to understand which way you prefer to use, in order to get some ETH into your account (I recommend going on the Gitter chat and requesting it).</p> <p>That’s it! After starting up and some ETH in your account, you can deploy your ICO contracts to a blockchain:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn deploy</span> </span></code></pre> <p><strong>Note #1</strong>, check your Parity UI for any approval request during deploy (and approve it).</p> <p><strong>Note #2</strong>, if you’re getting this error:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">Error: The contract code couldn&#39;t be stored, please check your gas amount. </span></code></pre> <p>You need to increase your amount of GAS (edit the popup that opens in Parity UI to approve the request).</p> <p><strong>Note #3</strong>, if you’re having too much problems with the Kovan network (no peers, slow deploy/neverending deploy, can’t get ETH into your account), you can use the Ropsten network (get ETH for Ropsten <a href="http://faucet.ropsten.be:3001/">here</a>):</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> parity ui<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>chain</span> ropsten</span> </span></code></pre> <p>After deploy finishes you’ll have a Crowdsale and Token addresses in the deploy output.</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">Deploying CooliumCrowdsale... </span><span class="z-text z-plain">... 0x685c094ac05f2f3fc35754cae56cba54b0c1b7f4be1f7765b9dfb936bec632bf </span><span class="z-text z-plain">CooliumCrowdsale: 0x4e72770760c011647d4873f60a3cf6cdea896cd8 </span><span class="z-text z-plain">Saving successful migration to network... </span><span class="z-text z-plain">... 0x2613896914f5d1eeaed6be42e3e0466d891206e27c197a7bac0078225f4d693a </span><span class="z-text z-plain">CooliumToken: 0xcee94e5d4c424e229af969aa1c1fd0e1a9de9adb </span></code></pre> <p>In order to test participating in your ICO:</p> <ul> <li>Remember this is all on the testnet so configure your client to it (you can use Metamask to send to Kovan/etc.)</li> <li>Make sure the ICO started (remember the start time you set?).</li> <li>Make sure people send ETH to the Crowdsale address.</li> <li>For production ICO, you need to repeat this process with the Ethereum blockchain, which means you’ll need real ETH funds, etc. (out of scope).</li> </ul> <h3 id="recommended-tools-and-libraries">Recommended Tools and Libraries</h3> <p>If you’d like to use the code you generated as a starting point, and keep on developing for the Ethereum blockchain, you might want to get familiar with a few more things.</p> <ul> <li>Install the <a href="https://github.com/juanfranblanco/vscode-solidity">solidity</a> vscode extension for linting, errors and general IDE assists.</li> <li>We use <a href="https://github.com/trufflesuite/truffle">truffle</a> as an Ethereum development framework. At the time of writing there’s <em>very</em> little reason to look elsewhere (and there are a few alternatives).</li> <li>We use <a href="https://github.com/OpenZeppelin/zeppelin-solidity">Zeppelin</a> which is quickly becoming an industry standard for writing secure Solidity contracts and it also has blueprints for common tasks. In fact all of the complexity of our contracts (and all of the credit) goes there.</li> <li>Use <a href="https://metamask.io">Metamask</a> to interact with your crowd sale. It’ll be nicer overall.</li> <li>We use <a href="http://www.hygen.io">Hygen</a> to automate the entire coding process.</li> <li>We use <a href="https://github.com/paritytech/parity">Parity</a> to work against an Ethereum blockchain.</li> <li>We use <a href="https://github.com/trufflesuite/ganache">Ganache</a> for faster workflow and testing when deploying.</li> </ul> <h3 id="conclusion">Conclusion</h3> <p>This little walkthrough demonstrated how easy it is to build a crowdsale for an ICO from an Ethereum contract programming point of view. We did it on the fly, with <a href="https://github.com/trufflesuite/truffle">Truffle</a>, <a href="https://github.com/OpenZeppelin/zeppelin-solidity">Zeppelin</a> and we made it fun with <a href="http://www.hygen.io">Hygen</a>.</p> <p>Obviously, if you’re really planning an ICO — the devil is in the details, so don’t just blindly take whatever that’s here and use it. You can, however, use it as a starting point and as a way to understand how you want your ICO to work.</p> Build faster by generating code with Hygen.io Wed, 24 Jan 2018 00:00:00 +0000 Unknown https://rng0.io/blog/build-faster-by-generating-code-with-hygen-io/ https://rng0.io/blog/build-faster-by-generating-code-with-hygen-io/ <p><img src="https://rng0.io/blog/build-faster-by-generating-code-with-hygen-io/1.png" alt="" /></p> <p>The scalable code generator that lives in your project.</p> <p><a href="http://hygen.io">hygen</a> is a code generator that lives in your project and scales with your team. It's fast, flexible, and perfectly fits modular code bases.</p> <p>For the impatient, you check out the <a href="http://www.hygen.io/quick-start">documentation</a> and how to use it with <a href="http://www.hygen.io/redux">Redux</a>, <a href="http://www.hygen.io/react-native">React Native</a>, and <a href="http://www.hygen.io/express">Express.js</a>.</p> <p>But if you’d like a story about software design as well, keep reading. The story is told by hijacking a couple of fundamental user experience design laws (well, they’re not <em>really</em> laws, but, yea).</p> <h2 id="hick-s-law">Hick’s Law</h2> <p><a href="https://en.wikipedia.org/wiki/Hick%27s_law">Hick’s law</a> states that:</p> <blockquote> <p>The time it takes for a person to make a decision is a result of the possible choices he or she has: increasing the number of choices will increase the decision time.</p> </blockquote> <p>Turning to technology, choosing a new tool is a massive uptake of new choices, and the process of <em>learning</em> that new tool amplifies it even more. Every step of the way, friction is the default.</p> <p>As far as code generators go, <code>hygen</code> has a primary design goal that is to minimize this friction. The rest — is bonus.</p> <p>Every code generator tool has a templating engine. But there’s also meta data; like where to place the new file at. To do metadata we placed a front-matter in the <em>same</em> template file, just like <a href="https://jekyllrb.com/">Jekyll</a> does, and every Jekyll inspired blog generator that surfaced over the last few years (there’s plenty!).</p> <p>If you have your own static blog generated with Jekyll, or any other tool that’s inspired by it, then this should be super familiar.</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">--- </span><span class="z-text z-plain">to: hygen-examples/mailers/&lt;%= message || &#39;unnamed&#39;%&gt;/html.ejs </span><span class="z-text z-plain">inject: true </span><span class="z-text z-plain">before: &quot;const modules = [&quot; </span><span class="z-text z-plain">skip_if: newModule </span><span class="z-text z-plain">--- </span></code></pre> <p>To render, we use <code>ejs</code> - a ubiquitous, non restricted (as opposed to logic-less) templating engine. We don’t deal with the logic-less flamewar. If you want <a href="https://www.ebayinc.com/stories/blogs/tech/the-case-against-logic-less-templates/">logic in your templates</a>, go ahead; we trust that you're responsible. There's plenty of ways to shoot your own foot but above all we want to be pragmatic.</p> <p>From a syntax point of view a variant of <code>ejs</code> <a href="https://ruby-doc.org/stdlib-2.5.0/libdoc/erb/rdoc/ERB.html">probably</a> <a href="https://en.wikipedia.org/wiki/PHP">exists</a> in <a href="http://www.makotemplates.org/">what ever</a> language you're coming from.</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">View</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>./view<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">storiesOf</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>&lt;%= Name %&gt;<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-support z-type z-object z-module z-ts">module</span><span class="z-meta z-brace z-round z-ts">)</span> </span><span class="z-source z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">addDecorator</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">withKnobs</span><span class="z-meta z-brace z-round z-ts">)</span> </span><span class="z-source z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">addDecorator</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">withTests</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>&lt;%= Name %&gt;<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span> </span><span class="z-source z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">add</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>default<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span><span class="z-cast z-expr z-ts"> <span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">View</span> /<span class="z-meta z-brace z-angle z-ts">&gt;</span></span></span><span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <p>And we take a batteries included approach; a set of built-in commands to guide you to the next step.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> hygen init self</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Loaded</span></span><span class="z-meta z-function-call z-arguments z-shell"> templates: src/templates</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">added:</span></span><span class="z-meta z-function-call z-arguments z-shell"> _templates/generator/with-prompt/hello.ejs.t</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">added:</span></span><span class="z-meta z-function-call z-arguments z-shell"> _templates/generator/with-prompt/prompt.ejs.t</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">added:</span></span><span class="z-meta z-function-call z-arguments z-shell"> _templates/generator/new/hello.ejs.t</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> hygen generator new<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>name</span> mygen</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Loaded</span></span><span class="z-meta z-function-call z-arguments z-shell"> templates: _templates</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">added:</span></span><span class="z-meta z-function-call z-arguments z-shell"> _templates/mygen/new/hello.ejs.t</span> </span></code></pre> <p>There’s also intuitive argument parsing.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> hygen module new<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>name</span> auth<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>tokens</span> bcrypt</span> </span></code></pre> <p>And fantastic prompts, courtesy of <a href="https://github.com/SBoudrias/Inquirer.js">inquirer</a>.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> hygen database setup<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>host</span> localhost</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">?</span></span><span class="z-meta z-function-call z-arguments z-shell"> Select database (Press <span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>space<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> to select, <span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>a<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> to toggle all, <span class="z-keyword z-operator z-assignment z-redirection z-shell">&lt;</span>i<span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> to invert selection</span><span class="z-meta z-function-call z-shell"></span>) </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">❯◯</span></span><span class="z-meta z-function-call z-arguments z-shell"> Postgres</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">◉</span></span><span class="z-meta z-function-call z-arguments z-shell"> sqlite</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">◯</span></span><span class="z-meta z-function-call z-arguments z-shell"> MySQL</span> </span></code></pre> <p><code>hygen</code> doesn't impose on you a new programming language, project setup, or workflow. Because it lives <em>in</em> your project, and <em>in</em> your existing workflow and setup, it just inherits those.</p> <h2 id="parkinson-s-law">Parkinson’s Law</h2> <p>Let’s hijack <a href="https://en.wikipedia.org/wiki/Parkinson%27s_law">Parkinson’s law</a> for a moment. This is <em>the</em> problem with every generator framework I know.</p> <blockquote> <p>Work expands so as to fill all of the time available for its completion.</p> </blockquote> <p>When you have code generation set up as a separate project in or out of your repo (say, with <a href="https://github.com/audreyr/cookiecutter">cookiecutter</a> or <a href="https://github.com/yeoman/yo">yeoman</a>), it will become <em>a thing</em>. A shiny new toy.</p> <p>At best, it becomes a product you look after that throws you out of context every time you want to tweak a template. At worst, it becomes stale and unmaintained — work invested, and value never extracted.</p> <h3 id="rails-did-it-first">Rails Did It First</h3> <p>Wholesale project generators are still good for when you need an entire starter-project, although I doubt will survive given the surge of starter projects for which you can just <code>git clone</code> and move on.</p> <p>They’re also less great if you want to embed a generator workflow into your existing project, like <a href="http://guides.rubyonrails.org/command_line.html#rails-generate">Rails</a> did when <em>it</em> kickstarted the idea of having generators be a core part of developer productivity into mass adoption for the first time.</p> <pre data-lang="ruby" class="language-ruby z-code"><code class="language-ruby" data-lang="ruby"><span class="z-source z-ruby"><span class="z-meta z-class z-ruby"><span class="z-keyword z-control z-class z-ruby">class</span></span><span class="z-meta z-class z-ruby"> </span><span class="z-meta z-class z-ruby"><span class="z-entity z-name z-class z-ruby">CreateMigration</span></span><span class="z-meta z-class z-ruby"> <span class="z-punctuation z-separator z-inheritance z-ruby">&lt;</span></span><span class="z-meta z-class z-ruby"> </span><span class="z-meta z-class z-ruby"><span class="z-entity z-other z-inherited-class z-ruby"><span class="z-support z-other z-namespace z-ruby">Thor</span><span class="z-punctuation z-accessor z-double-colon z-ruby">::</span><span class="z-support z-other z-namespace z-ruby">Actions</span><span class="z-punctuation z-accessor z-double-colon z-ruby">::</span>CreateFile</span></span> </span><span class="z-source z-ruby"> <span class="z-meta z-function z-ruby"><span class="z-keyword z-control z-def z-ruby">def</span></span><span class="z-meta z-function z-ruby"> <span class="z-entity z-name z-function z-ruby">migration_dir</span></span> </span><span class="z-source z-ruby"> <span class="z-support z-class z-ruby">File</span><span class="z-punctuation z-accessor z-dot z-ruby">.</span>dirname<span class="z-punctuation z-definition z-group z-begin z-ruby">(</span><span class="z-variable z-other z-readwrite z-instance z-ruby"><span class="z-punctuation z-definition z-variable z-ruby">@</span>destination</span><span class="z-punctuation z-definition z-group z-end z-ruby">)</span> </span><span class="z-source z-ruby"> <span class="z-keyword z-control z-ruby">end</span> </span><span class="z-source z-ruby"> </span><span class="z-source z-ruby"> <span class="z-meta z-function z-ruby"><span class="z-keyword z-control z-def z-ruby">def</span></span><span class="z-meta z-function z-ruby"> <span class="z-entity z-name z-function z-ruby">migration_file_name</span></span> </span><span class="z-source z-ruby"> <span class="z-variable z-other z-readwrite z-instance z-ruby"><span class="z-punctuation z-definition z-variable z-ruby">@</span>base</span><span class="z-punctuation z-accessor z-dot z-ruby">.</span>migration_file_name </span><span class="z-source z-ruby"> <span class="z-keyword z-control z-ruby">end</span> </span><span class="z-source z-ruby"> </span><span class="z-source z-ruby"> <span class="z-meta z-function z-ruby"><span class="z-keyword z-control z-def z-ruby">def</span></span><span class="z-meta z-function z-ruby"> <span class="z-entity z-name z-function z-ruby">identical?</span></span> </span><span class="z-source z-ruby"> exists? <span class="z-keyword z-operator z-logical z-ruby">&amp;&amp;</span> <span class="z-support z-class z-ruby">File</span><span class="z-punctuation z-accessor z-dot z-ruby">.</span>binread<span class="z-punctuation z-definition z-group z-begin z-ruby">(</span>existing_migration<span class="z-punctuation z-definition z-group z-end z-ruby">)</span> <span class="z-keyword z-operator z-comparison z-ruby">==</span> render </span><span class="z-source z-ruby"> <span class="z-keyword z-control z-ruby">end</span> </span><span class="z-source z-ruby"><span class="z-keyword z-operator z-range z-ruby">...</span> </span></code></pre> <p>Rails and <a href="https://github.com/erikhuda/thor">Thor</a>, the generator framework it used, changed the way I thought about code generators. Up until then, when I needed to generate code (mostly for ORM entities, ah the times we had!) I was sucked up into <a href="https://en.wikipedia.org/wiki/Text_Template_Transformation_Toolkit">.NET’s T4 Text Templates</a>.</p> <p>That was 7 years ago. I kept making generators with Thor and friends hoping for productivity spikes. But these became projects I maintained which sucked up valuable time, counter-intuitively — they were lowering the returns in general productivity.</p> <h2 id="the-shopping-list">The Shopping List</h2> <p>That only means one thing. A shopping list for a generator framework that stays out of your way.</p> <ul> <li>Process ergonomics — I don’t want to compile my generators, or have them on a CI pipeline of their own.</li> <li>Developer ergonomics — easily accessible and easily invoked.</li> <li>Low friction — the <em>pitfall of success</em>. Each step I take should lead me to the next.</li> <li>Technology agnostic — don’t want a new tech stack.</li> <li>Contextual — if I’m on the data layer, I want data generators.</li> <li>Scalable — should work for multiple teams iterating over a large and modular codebase.</li> <li>Feature packed — simple design doesn’t mean a poor number of features.</li> <li>Flexible — give me a way to shoot my foot if I want to.</li> <li>Embeddable — can be composed into other projects.</li> <li>Super customizable — defaults are OK, but give me escape hatches.</li> <li>Clean, intentful — first be useful, only then be cool.</li> </ul> <p>That’s what <code>hygen</code> became.</p> <h2 id="hygen">Hygen</h2> <p><img src="https://rng0.io/blog/build-faster-by-generating-code-with-hygen-io/2.gif" alt="" /></p> <p>You can use it right now, in what ever project you have open. Here’s how to make a generator that adds a markdown document to your <code>docs/</code> folder.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> npm i<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>g</span> hygen</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cd your-project</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> hygen init self</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> hygen generator new<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>name</span> docs</span> </span></code></pre> <p>Edit <code>_templates/docs/new/hello.ejs.t</code>:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">--- </span><span class="z-text z-plain">to: docs/&lt;%= name %&gt;.md </span><span class="z-text z-plain">--- </span><span class="z-text z-plain"> </span><span class="z-text z-plain">Hi! </span><span class="z-text z-plain">I&#39;m a document about &lt;%= name %&gt; </span></code></pre> <p>And then rename. The name of the file doesn’t matter, it’s for you, for bookeeping purposes.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> mv _templates/docs/new/<span class="z-meta z-group z-expansion z-brace z-shell"><span class="z-punctuation z-section z-expansion z-brace z-begin z-shell">{</span>hello<span class="z-punctuation z-separator z-shell">,</span>new-doc<span class="z-punctuation z-section z-expansion z-brace z-end z-shell">}</span></span>.ejs.t</span> </span></code></pre> <p>That’s it! Let’s make a doc:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> hygen docs new<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>name</span> architecture</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Loaded</span></span><span class="z-meta z-function-call z-arguments z-shell"> templates: _templates</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">added:</span></span><span class="z-meta z-function-call z-arguments z-shell"> docs/architecture.md</span> </span></code></pre> <p>And now let’s check our new generator in:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">❯ gs </span><span class="z-text z-plain">A _templates/docs/new/new-doc.ejs.t </span><span class="z-text z-plain">A _templates/generator/new/hello.ejs.t </span><span class="z-text z-plain">A _templates/generator/with-prompt/hello.ejs.t </span><span class="z-text z-plain">A _templates/generator/with-prompt/prompt.ejs.t </span></code></pre> <p><code>_templates</code> is a <em>contextual folder</em>. <code>hygen</code> looks for one, at the folder you’re running it from. This way you can have different flavors of the same generator happen from different parts of your project, or maybe in a mono-repo scenario just different teams having their own sets of generators based on where in the mono-repo a team works at.</p> <p>Note that the <code>hygen generator new</code> command is <em>also</em> checked in (the <code>generator/new</code> and <code>generator/with-prompt</code> parts). This is part of the "flexibility" principle.</p> <p>We acknowledge that it may be that you don't like the vanilla <code>hygen</code> generator for new generators. Go ahead and change it; then check it in and have your team review it. After this change all new generators you build will incorporate <em>your</em> new wayto make them.</p> <h2 id="power-to-the-generator">Power to the Generator</h2> <p><code>hygen</code> lets you build generators that add multiple files, make injections to existing files, have prompts for interactivity with the user, and more.</p> <p>Here’s how to ease some Redux boilerplate fatigue.</p> <p>These days I choose <a href="https://github.com/erikras/ducks-modular-redux">ducks</a>, to remove a little bit of boilerplate and have modularity baked in any app I build.</p> <p>My typical Redux architecture would look like this:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">app/ </span><span class="z-text z-plain"> components/ </span><span class="z-text z-plain"> icon.js </span><span class="z-text z-plain"> avatar.js </span><span class="z-text z-plain"> modules/ </span><span class="z-text z-plain"> boot.js &lt;---- glues modules together, requires chat, app, and auth. </span><span class="z-text z-plain"> chat/ </span><span class="z-text z-plain"> index.js &lt;---- the &#39;connect&#39; bit for Redux. </span><span class="z-text z-plain"> view.js &lt;---- the view, separated, for testing. </span><span class="z-text z-plain"> state.js &lt;---- reducer, actions, types, selectors. </span><span class="z-text z-plain"> app/ </span><span class="z-text z-plain"> index.js </span><span class="z-text z-plain"> view.js </span><span class="z-text z-plain"> state.js </span><span class="z-text z-plain"> auth/ </span><span class="z-text z-plain"> index.js </span><span class="z-text z-plain"> view.js </span><span class="z-text z-plain"> state.js </span></code></pre> <h2 id="adding-a-module">Adding a Module</h2> <p>Adding a new module is very easy with <code>hygen</code>. Here's how your templates look like:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">_templates/ </span><span class="z-text z-plain"> module/ </span><span class="z-text z-plain"> new/ </span><span class="z-text z-plain"> index.ejs.t </span><span class="z-text z-plain"> view.ejs.t </span><span class="z-text z-plain"> state.ejs.t </span><span class="z-text z-plain"> inject_boot.ejs.t &lt;--- adds a &#39;require&#39; clause to boot.js </span></code></pre> <p>Here’s how <code>index</code> looks like:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">--- </span><span class="z-text z-plain">to: app/modules/&lt;%= name %&gt;/index.js </span><span class="z-text z-plain">--- </span><span class="z-text z-plain">// </span><span class="z-text z-plain">// requires, mappings, etc.... </span><span class="z-text z-plain">// </span><span class="z-text z-plain">export default connect(...)(&lt;%= Name %&gt;) </span></code></pre> <p>A similar trick would do for <code>view</code> and <code>state</code>.</p> <p>How would we add a require line given that <code>boot.js</code> looks like this?</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> ... some bootstrapping code ...</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">modules</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> &lt;--- we want to inject a line here!</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-support z-function z-ts">require</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>chat<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">default</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-support z-function z-ts">require</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>auth<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">default</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-support z-function z-ts">require</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>app<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">default</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">]</span></span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> ... rest of bootstrapping code ...</span> </span></code></pre> <p>Let’s build <code>inject_boot</code>:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">--- </span><span class="z-text z-plain">to: app/modules/boot.js </span><span class="z-text z-plain">inject: true </span><span class="z-text z-plain">skip_if: &lt;%= name %&gt; </span><span class="z-text z-plain">after: &quot;const modules = [&quot; </span><span class="z-text z-plain">--- </span><span class="z-text z-plain">require(&#39;./&lt;%= name %&gt;).default, </span></code></pre> <p>And we’re done! Generating a new module is saying this:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> hygen module new<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>name</span> settings</span> </span></code></pre> <h2 id="use-hygen-today">Use Hygen Today</h2> <p><code>hygen</code> isn’t just for Redux or documentation, or the other use cases that we line out in the documentation. It’s for everything, because it imposes almost nothing.</p> <p>There’s more to see on <a href="http://hygen.io">hygen.io</a> where the documentation lives at. But if you want a more gentle intro there’s also the digest form in the <a href="http://github.com/jondot/hygen">README</a> <em>and</em> <code>hygen</code> is built with zero configuration and no strings attached so no harm done trying it.</p> <p>So if you like trying things first, just install it, make a few generators and see if it fits you.</p> 10 React Developer Tools You Might Have Missed Wed, 10 Jan 2018 00:00:00 +0000 Unknown https://rng0.io/blog/10-react-developer-tools-you-might-have-missed/ https://rng0.io/blog/10-react-developer-tools-you-might-have-missed/ <p>Here’s a few tools you might have missed, out of my React tooling shortlist that deal with performance, visibility and debugging. Note that some of the more generic, less DOM dependent tools apply to React on React Native as well.</p> <h3 id="react-sight">React Sight</h3> <p><img src="https://rng0.io/blog/10-react-developer-tools-you-might-have-missed/1.gif" alt="" /></p> <p><a href="https://github.com/React-Sight/React-Sight">React Sight</a> is a React visualization tool with a live state and prop tree viewer that complements React devtools.</p> <h4 id="reselect-devtools">Reselect Devtools</h4> <p><img src="https://rng0.io/blog/10-react-developer-tools-you-might-have-missed/2.png" alt="" /></p> <p><a href="https://github.com/skortchmark9/reselect-tools">Reselect devtools</a> is a visualization tool for <a href="https://github.com/reactjs/reselect">reselect</a> which you can use to follow the caching layers.</p> <h4 id="redux-devtools-profiler-monitor">Redux DevTools Profiler Monitor</h4> <p><img src="https://rng0.io/blog/10-react-developer-tools-you-might-have-missed/3.jpg" alt="" /></p> <p><a href="https://github.com/pbomb/redux-devtools-profiler-monitor">Redux DevTools Profiler Monitor</a> a cool way to start and stop a Javascript profile session, based on Redux action.</p> <h4 id="periscope">Periscope</h4> <p><img src="https://rng0.io/blog/10-react-developer-tools-you-might-have-missed/4.jpg" alt="" /></p> <p><a href="https://github.com/shea-hawkins/periscope">Periscope</a> is a timeline based monitoring for Redux actions.</p> <h4 id="redux-promise-middleware-times">redux-promise-middleware-times</h4> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-variable z-other z-readwrite z-ts">mport</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-variable z-other z-readwrite z-ts">createStore</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">applyMiddleware</span> <span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-variable z-other z-readwrite z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">thunk</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux-thunk<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">promiseMiddleware</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux-promise-middleware<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">logTimes</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux-promise-middleware-times<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">store</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">createStore</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">thunk</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">promiseMiddleware</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">logTimes</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <p><a href="https://github.com/lemonleon/redux-promise-middleware-times">redux-promise-middleware-times</a> is a tiny addon to <a href="https://github.com/pburtchaell/redux-promise-middleware">redux-promise-middleware</a> that records time it took for async actions to happen.</p> <h4 id="redux-duplicate-actions">redux-duplicate-actions</h4> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">__DEV__</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> To throw a fatal error use the below:</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">reduxDuplicateActions</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-function z-ts">require</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux-duplicate-actions<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-constant z-language z-boolean z-true z-ts">true</span><span class="z-meta z-brace z-round z-ts">)</span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">reduxMiddleware</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">push</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">reduxDuplicateActions</span><span class="z-meta z-brace z-round z-ts">)</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> To just show a warning:</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">reduxDuplicateActions</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-function z-ts">require</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux-duplicate-actions<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-constant z-language z-boolean z-false z-ts">false</span><span class="z-meta z-brace z-round z-ts">)</span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">reduxMiddleware</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">push</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">reduxDuplicateActions</span><span class="z-meta z-brace z-round z-ts">)</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p><a href="https://github.com/daviemakz/redux-duplicate-actions">redux-duplicate-actions</a> detects duplicate actions (glitches in clicks, double action bugs, etc.) along the redux action stream.</p> <h4 id="should-component-update-dev">should-component-update-dev</h4> <p><img src="https://rng0.io/blog/10-react-developer-tools-you-might-have-missed/5.jpg" alt="" /></p> <p><a href="https://github.com/ConciergeAuctions/should-component-update-dev">should-component-update-dev</a> is another take on <a href="https://github.com/maicki/why-did-you-update">why-did-you-update</a> where the integration point is swapping a component’s <code>shouldComponentUpdate</code>.</p> <h4 id="react-wastage-monitor">react-wastage-monitor</h4> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">React</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>react<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">ReactDOM</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>react-dom<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">ReactWastageMonitor</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>react-wastage-monitor<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-support z-variable z-object z-process z-ts">process</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-process z-ts">env</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-constant z-property z-ts">NODE_ENV</span> <span class="z-keyword z-operator z-comparison z-ts">!==</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>production<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">ReactWastageMonitor</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">React</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">ReactDOM</span><span class="z-meta z-brace z-round z-ts">)</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p><a href="https://github.com/MalucoMarinero/react-wastage-monitor">react-wastage-monitor</a> is yet another take on <a href="https://github.com/maicki/why-did-you-update">why-did-you-update</a>, where there’s support to detect whether DOM updated or not.</p> <h4 id="react-monocle">React Monocle</h4> <p><img src="https://rng0.io/blog/10-react-developer-tools-you-might-have-missed/6.gif" alt="" /></p> <p><a href="https://github.com/team-gryff/react-monocle">react-monocle</a> is a powerful React visualization that takes the source code components, runtime components and their state, and your redux state and fuses all together.</p> <h4 id="react-component-benchmark">react-component-benchmark</h4> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">Benchmark</span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">component</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">MyComponent</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">componentProps</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">componentProps</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">onComplete</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>this.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">_handleComplete</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">ref</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>this.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">_setBenchRef</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">samples</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-constant z-numeric z-decimal z-ts">50</span></span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">timeout</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-constant z-numeric z-decimal z-ts">10000</span></span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">type</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>BenchmarkType.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">MOUNT</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span><span class="z-source z-ts"> <span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></code></pre> <p><a href="https://github.com/paularmstrong/react-component-benchmark">react-component-benchmark</a> is a convenient and declarative component-based benchmarking tool that you can use to benchmark components and entire trees of components. Also check out <a href="https://github.com/apiv/fuego">fuego</a> which seems to be the same thing with a different API.</p> <h2 id="one-more-thing">One more thing</h2> <p><img src="https://rng0.io/blog/10-react-developer-tools-you-might-have-missed/7.png" alt="" /></p> <p><a href="https://github.com/jondot/hygen">Hygen is a code generator framework</a> you can use in any project to reduce boilerplate. This is painfully evident for React and Redux, where the amount of boilerplate (for good reasons) can be overwhelming. For example, you can generate a component, a test, a storybook, and documentation every time you want a new component:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> hygen component new<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>name</span> avatar</span> </span></code></pre> Analyzing Markets at Scale — A Crawling Blueprint Sun, 29 Oct 2017 00:00:00 +0000 Unknown https://rng0.io/blog/analyzing-markets-at-scale-a-crawling-blueprint/ https://rng0.io/blog/analyzing-markets-at-scale-a-crawling-blueprint/ <p>Crawling and scraping sounds like this one-time flaky quirky work. But what kind of products or companies need this for their core offering?</p> <ul> <li>Price tracking and comparative websites like <a href="https://www.newegg.com/">newegg</a>.</li> <li>Search engine optimization (SEO) and app store optimization (ASO) products.</li> <li>Market analysis and optimization companies, like <a href="https://www.similarweb.com/">SimilarWeb</a> and <a href="https://www.appannie.com">AppAnnie</a>.</li> </ul> <p>Building for this kind of scale needs reliable and scalable crawling and data architectures for its core offering. These requirements reveal the following, sometimes conflicting, challenges.</p> <ol> <li><a href="https://en.wikipedia.org/wiki/Information_retrieval"><strong>Information retrieval</strong></a>. The source of our data are websites, but not limited to that. It can also be PDFs, spreadsheets, and APIs. These sources can vary in the degree of friendliness towards us.</li> <li><a href="https://en.wikipedia.org/wiki/Natural_language_processing"><strong>Natural language processing</strong></a>. Data normalization, and the semantic web. Wide fields of research, but we still have to do a bit of each when presented with painfully unstructured pieces of data.</li> <li><strong>Networking</strong>. Traffic routes and counter-protection. Crawling and scraping involves taking in a big and complete amount of data in a short amount of time, sometimes targeting resources that aren’t built for it or (without us taking a side in the legal aspects of it) not interested in it happening, for reasons such as terms of service and more.</li> <li><strong>Big data pipelines, aggregation and analytics</strong>. With almost any crawling infrastructure, we’re bound to pile an ever-increasing, large, amount of data with every job and every time unit (hours, days, weeks).</li> <li><strong>Speed layer and search</strong>. Almost all crawling solutions need some kind of speed layer. One that can provide answers for a subset of the questions in real-time, rather answering <em>all</em> questions after a long while.</li> <li><strong>Agility</strong>. Our target data sources change (read: websites), our auxiliary data sources change (read: exchange rates), and our requirements change (read: features). This kind of multiplier of change requires an agile turn-around operation to compensate throughout crawling, data pipeline, and search (read: rebuild, rerun, reindex).</li> </ol> <p>And a ton more. You’d be surprised at how many products you use every day that build on a core set of data that’s they source from crawling, and that fact is swept to the back of the stage, because crawling, data cleansing, and flakyness is not cool and no longer as marketable as it was in the past.</p> <h2 id="general-architecture">General Architecture</h2> <p>We’re going to deep dive from a general architecture for crawling and data processing that deals with a crawler, a batch layer and a query layer like this:</p> <p><img src="https://rng0.io/blog/analyzing-markets-at-scale-a-crawling-blueprint/1.png" alt="" /></p> <p>Into something more prescriptive that you can actually use, naming the components and tools that I use in production, like this:</p> <p><img src="https://rng0.io/blog/analyzing-markets-at-scale-a-crawling-blueprint/2.png" alt="" /></p> <p>Here’s how the flow works.</p> <ol> <li>Scheduler (Airflow) fires off a new crawl job, specifying the current time unit for this job. It may fire off several of these jobs for redundancy, as crawling is flaky; accepting that we’ll have duplicate data from the get-go. We’ll also start auxiliary crawls to enrich our data, such as fetching exchange rates from dedicated brokers. The scheduler is also responsible to coordinate and converge the jobs it started as a graph.</li> <li>For each target, our crawler goes out through our proxy (8), fuzzing out a request IP, user agent, data center and so on, and handing back a raw HTML content (for when the target source is a website).</li> <li>Each candidate web page goes through an internal crawler pipeline which does a subset of the data manipulation needed: deduping, normalization, item tracking and item export.</li> <li>At the tracking stage, we drive items towards a cache to make sure to drop duplicates, and onto a search engine, available for query in real-time.</li> <li>Items piled up in data feeds on S3 as the job progresses. When done, we’ll have a bunch of JSON files for a main crawl and its redundancy crawls as well as any auxiliary crawl that we fired off.</li> <li>Finally, Airflow (1) starts a data flow job, using <a href="https://airflow.incubator.apache.org/code.html?highlight=sensors#airflow.operators.sensors.BaseSensorOperator">sensors</a> to detect when data is available.</li> <li>We load job data results onto the indexing &amp; tracking component (4) for a most-up-to-date view of the world, using a relational database such as Postgres, to provide an OLAP style querying solution.</li> </ol> <p>Let’s have a deep dive into every component, and put some more content behind what this architecture means practically.</p> <h2 id="crawler">Crawler</h2> <p><a href="http://scrapy.readthedocs.io">Scrapy</a> makes an agile and flexible crawling and scraping platform. It provides the quickest turn around from an experiment to a production project. Still, it’s not a broad-crawl platform like <a href="http://nutch.apache.org/">Nutch</a> is, and the mindset is different.</p> <p>In the Scrapy mindset you crawl and scrape, extract content in one go, patch up any difficulties you might have <em>while</em> crawling, including any logic for coalescing or cleansing data types and navigating the target site more effectively; for example, normalizing product prices or making sure Javascript fragments get rendered before scraping.</p> <p>In the Nutch mindset, you download a chunk of the internet. You then perform a big-data pass on the raw results to form structured data. There’s flexibility in crawl time, should you want to be more agile, but it’s done through <a href="https://wiki.apache.org/nutch/PluginCentral">configuration and plugin setup</a> rather than code.</p> <p>If you’re new to Scrapy, I recommend looking at a <a href="https://github.com/scrapy/quotesbot">sample crawler</a> and a more <a href="https://github.com/scrapinghub/portia/tree/master/slybot/slybot">realistic crawler</a> to get the gist of it; it’s self explanatory because Scrapy does the heavy lifting for you. What’s less obvious to newcomers is the extensibility points and plugins Scrapy comes with and how does a fully rigged solution look like, so let’s take a look at some of the more popular components around it.</p> <p>Scrapy revolves around simple concepts.</p> <ol> <li><strong>Spider</strong>. What ever is doing the crawl, maintains the crawl frontier, lifecycle, moves pages through your code, and then onto the <em>pipeline</em>.</li> <li><strong>Pipeline</strong>. All discovered items go through the pipeline; which can be an empty one, or one that flushes items to files by format — CSV, JSON or what have you.</li> <li><strong>Middleware</strong>. Pieces of code that wake up during the different stages in the spider’s lifecycle. One example is a middleware that will <a href="https://github.com/alecxe/scrapy-fake-useragent">randomly swap user agents</a> for each external request.</li> <li><strong>Configuration</strong>. On first glance sounds pretty static and trivial, but here it behaves much like feature flags for your crawler since your configuration file is code. Some ideas are to activate or disable your pieces of code per variant of a crawl, swap a piece of infrastructure (anonymising proxy), or perform staging and test runs if you’d like — all by remotely/locally specifying flags per run.</li> </ol> <h3 id="delta-crawls">Delta Crawls</h3> <p>A <em>delta crawl</em> is a crawl process that will output new or changed items, since the last <em>crawl</em> which makes a smaller incremental addition to our data set. To do that, Scrapy has to store some kind of state.</p> <ul> <li>With <a href="https://github.com/jondot/pgpipeline">pgpipeline</a> we use a database for book keeping; here Postgres but most relational databases Python supports work. While <code>pgpipeline</code> does more than modeling delta crawls (more on that later), it supports being able to store new/modified items to your database automatically.</li> <li>With <a href="https://github.com/scrapy-plugins/scrapy-deltafetch">deltafetch</a> the book keeping is at the page level; ignoring pages that we saw in previous crawls. A less flexible solution than <code>pgpipeline</code> but still works for modeling delta crawls on a file system, which everybody has.</li> </ul> <h3 id="throttling-anti-crawler-bypass">Throttling/Anti-crawler Bypass</h3> <p>If you’re trying to bypass someone’s protection mechanisms or terms of service — that can be illegal to varying levels of degrees.</p> <p>I’ll assume we have some kind of funny situation where we’re crawling against your own company, let’s say for security or compliance, and, say, that this other team that’s responsible for the target website cannot lift their protection mechanisms for you or don’t have the time to build something that conditionally does it for you.</p> <p>There, we cleared our conscious.</p> <p>For bypassing crawler-circumventing mechanisms, one sure way is to use (without promoting any of those) either of the following plugins, to make sure requests fan-out through different IPs.</p> <ul> <li><a href="https://github.com/scrapy-plugins/scrapy-crawlera">crawlera</a> — anonymising proxy.</li> <li><a href="https://github.com/mizhgun/scrapy-proxymesh">proxymesh</a> — anonymising proxy.</li> <li><a href="https://github.com/cvibhagool/scrapy-tor">tor</a> — anonymising proxy, using <a href="https://www.torproject.org">the Onion Network</a>.</li> <li><a href="https://github.com/aivarsk/scrapy-proxies">open proxies</a>, and probably more to make sure the crawler agent’s IP gets changed every now and then. From experience that’s the most effective way to thwart most protection schemes.</li> <li><a href="https://github.com/alecxe/scrapy-fake-useragent">User agent faking and swapping</a> is also useful to avoid blacklisted user agents.</li> </ul> <p>These reverse proxies can be <em>very</em> slow, as some providers can take minutes(!) of response times when swapping IPs and that some of the providers (e.g. ProxyMesh) were’nt built for crawling purposes with all the related implications (legal, performance, etc.).</p> <h3 id="alternate-rendering">Alternate Rendering</h3> <p>To render pages with active Javascript, we can use <a href="https://github.com/scrapy-plugins/scrapy-splash">Splash</a>. That said <a href="https://github.com/GoogleChrome/puppeteer">headless chrome API</a> was out earlier this year, which sparked a ton of out-of-band rendering libraries, a gold mine for crawlers.</p> <p>When we do Javascript renders, our crawls become slow because there’s a browser of some sort that needs to go out and fetch data from an API and then complete a page’s render. Depending on feasibility, it’s preferrable to do auxiliary crawls that target the Javascript API individually and merge this data as part of the merge job; leaving us in a simple and fast HTML render domain.</p> <h3 id="crawl-persistence">Crawl Persistence</h3> <p>Scrapy has a built-in ability to cache crawl sessions, and resume sessions. If enabled, a <code>.scrapy</code> folder will appear, and in it a pickled storage of the last crawl activity, which it will use for next crawls as a cache.</p> <p>Regardless, we can use the following to augment or replace this capability.</p> <ul> <li><a href="https://github.com/scrapy-plugins/scrapy-pagestorage">pagestorage</a> stores requests and responses.</li> <li><a href="https://github.com/scrapy-plugins/scrapy-dotpersistence">dotpersistence</a> will sync the Scrapy <code>.scrapy</code> folder to S3.</li> </ul> <h3 id="data-normlization">Data Normlization</h3> <p>Other than using any Python library out the for data normalization (and there’s a lot), we can use Scrapy plugins that streamline that into your pipeline. Here’s a couple useful ones.</p> <ul> <li><a href="https://github.com/scrapy-plugins/scrapy-magicfields">magicfields</a> allows us to add standard fields to a scraped item (timestamp, etc.).</li> <li><a href="https://github.com/scrapy-plugins/scrapy-querycleaner">querycleaner</a> helps cleaning a noisy item URL, so that two identical items under slightly different URLs with referrers and arbitrary query params are semantically the same.</li> </ul> <h3 id="notifications">Notifications</h3> <p>Scrapy do notifications via different channels out of the box (that includes SMTP). A job started, ended, failed and what were the statistics is useful to keep from one crawl to another. We can also find 3rd party integrations such as <a href="https://github.com/jondot/scrapy-mailgun">Mailgun</a> that is smooth to use with the simplicity of an API key and a couple configuration values.</p> <h3 id="storage-and-indexing">Storage and Indexing</h3> <p>Scrapy has plugins for <a href="https://github.com/acordiner/scrapy-dynamodb">dynamodb</a>, <a href="https://github.com/sebdah/scrapy-mongodb">mongodb</a>, <a href="https://github.com/knockrentals/scrapy-elasticsearch">Elastic Search</a> which is useful, and <a href="https://github.com/jondot/pgpipeline">pgpipeline</a> for automatic persistence to Postgresql (and probably more).</p> <p>We can also dump scraped items to <a href="https://github.com/ziplokk1/botopipeline">SQS</a> and from then get a worker to take items off the queue and persist as you’d prefer.</p> <h3 id="crawling-and-ci-cd">Crawling and CI/CD</h3> <p>Scrapy runs as its own process by default. But there are other ways, perhaps more creative, to repackage and run it.</p> <ul> <li>As an embedded Python codebase. We can embed your crawler in an existing program or service; for example, we can crawl on-demand triggered by some agent making a request, and provide crawl results to the requesting agent when done.</li> <li>We can build a worker farm, where each worker runs a spider of its own. One implementation can be Jenkins and Jenkins workers, or Docker and ECS (my preferred way).</li> </ul> <p>As soon as you’re able to dockerize your crawler, a compute unit that can run Docker and a place to dump results is all you need. I’ve ran the same crawlers on a home array of Raspberry Pi’s, on Docker Cloud, on ScrapingHub and on ECS and it was perfectly transparent.</p> <p><img src="https://rng0.io/blog/analyzing-markets-at-scale-a-crawling-blueprint/3.png" alt="" /></p> <p>To be most effective, bake a container with everything in it; Python, Scrapy, as many plugins as you like, and your spiders code and configuration. The moment the container is up, it should start crawling.</p> <p>Running specifically on AWS can be cheap and powerful (more on processing data later), and flexible for data integrations. Use ECS (a small instance will do, as this is an I/O bound workload) and dump data directly on S3.</p> <h3 id="proxy">Proxy</h3> <p>Go out to the world to fetch bytes through a proxy. Using an anonymising network — that’s already happening. Using a proxy on top of that, or without that can be beneficial.</p> <ul> <li><em>Caching and micro-caching</em>. Even with Scrapy’s local file-caching feature, an HTTP caching proxy will give us better and more familiar levels of control for caching, and allow horizontal scaling when several crawlers are involved.</li> <li><em>Blocking sites and re-routing at the proxy level</em>. Malfunctioning sites, network components will require our intervention. With a proxy sitting between a crawler and the world, we can remote flip a switch for a strategy to amend such problems.</li> <li><em>Rerouting through other proxies towards other data centers</em>. We might decide to flow part of the traffic through an anonymising network and some of it out directly, or through a bunch of anonymising networks.</li> </ul> <p>In any case, we also get visibility for free from most proxy products: how many requests succeeded, failed, in-flight, latency, size of payloads and more.</p> <h3 id="pipeline">Pipeline</h3> <p>The Scrapy pipeline is the place where we do deduping, normalization, diff tracking, exporting and more. It’s the place where extracted data flows through.</p> <p>Almost everything mentioned here can be done on raw data in batch much later. Which steps do we do in Scrapy’s pipeline and which steps do we save for the, later, offline run?</p> <p>First, do we want to have a data pass after crawling has finished? if we’re still small, we’d probably want to keep everything together and skip that amount of extra work; and that’s fine, Scrapy will let us do that. In that case everything mentioned can be done in the pipeline, during the crawl.</p> <p><img src="https://rng0.io/blog/analyzing-markets-at-scale-a-crawling-blueprint/4.png" alt="" /></p> <p>If we do have an offline step already planned or set up, then steps like deduping, normalization diff tracking will be perfect there. We won’t have to split that kind of logic across two process and perhaps codebases. When we’re looking at <em>all of the data</em> we’re also smarter and have more context to do these operations; and we can also move faster and trial our ideas on that data, without invoking any lengthy and flaky recrawl process (because they always <em>are</em> flaky).</p> <p>That said, there are a few tasks we do want to perform during the crawl in Scrapy’s pipeline. These are tasks that are time-sensitive; one example would be performing IP to location, calls to 3rd party APIs or data conversions such as exchange rates.</p> <h3 id="feeds">Feeds</h3> <p>When we crawl a website, we ultimately want to end up with a list of items. With Scrapy we can have that list automatically serialized and exported through the <a href="https://doc.scrapy.org/en/latest/topics/feed-exports.html"><em>feeds</em></a> mechanism; it can be JSON, CSV, or what have you, through plugins that set themselves up at the end of your pipeline.</p> <p>When picking a format, JSON is a good bet, although it is verbose. For ad-hoc crawls I tend to use CSV, which fits well with the rest of the data wrangling toolset existing in Python, like pandas. For production I always use JSON. For JSON — keep your objects flat, don’t nest.</p> <p>Either case, when we’re running multiple crawls that should converge on the same entity, such as a baseline crawl and multiple auxiliary enrichment crawls, make sure there’s already an ID, or a strong strategy to join the data from the auxiliary crawls onto the baseline crawl; we don’t want to look for how to massage our data fields to join on after we’ve realized it can’t be joined, risking dumping the data and recrawling everything.</p> <p>One other thing — always output more than needed. It’s best to include a field when unsure rather than optimize it out of the data feed.</p> <h3 id="item-indexing-and-tracking">Item Indexing and Tracking</h3> <p>Building a crawling solution, we’ll require a way to provide search. Search makes data accessible for slicing and dicing, and immediately accessible in terms of real-timeness. It’s a quick win to bypass all other heavy work we’re going to do in our batch layer; this is our speedy data solution for random access use cases.</p> <p>When we use <a href="https://www.elastic.co/">Elastic Search</a> to index our data, we get a pretty good API for free, and plenty of UI interfaces to use for our backoffice. Not to mention a bullet proof search product that scales with our data.</p> <p><img src="https://rng0.io/blog/analyzing-markets-at-scale-a-crawling-blueprint/5.png" alt="" /></p> <p>Inflight item indexing is another technique we should use. We want to prevent duplicate work <em>while</em> running a crawl, which means we must identify when an item was already crawled and effectively dedupe it. For this, we’ll need an extra state store. Redis makes a perfect match both in speed and features. We can store actual items via key/value, or use a statistical counter like <a href="https://en.wikipedia.org/wiki/HyperLogLog">HLL</a> which Redis also <a href="https://redis.io/commands/pfcount">has</a>.</p> <p>Lastly, we push aggregated data onto <a href="https://www.postgresql.org/">Postgres</a> for more traditional OLAP-like querying.</p> <h3 id="batch-layer">Batch Layer</h3> <p>When all crawls finish, we need to do some data crunching. Matching options for different scale are <a href="https://pandas.pydata.org">pandas</a> , <a href="https://petl.readthedocs.io">petl</a>, <a href="http://www.pytables.org/">pytables</a>, <a href="https://spark.apache.org/">Apache Spark</a>.</p> <p>The basic idea is that given a way to load crawl result feeds from <em>anywhere</em>, process using <a href="https://www.jstatsoft.org/article/view/v040i01/v40i01.pdf">split-apply-combine</a>, map/reduce, or SQL-like grouping and aggregation, and, dump back into <em>anywhere</em> (databases, files). We want the tooling we use to have a great coverage of inputs and outputs; from/to S3, databases or others, and great aggregation and data join abstractions including flexible or programmable logic for <a href="https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.fillna.html">prefilling</a> and merging values when joining multiple datasources (more on that later).</p> <h3 id="scheduling">Scheduling</h3> <p>We schedule two job types regularily: crawling jobs and data processing jobs. Ideally one is upstream to the other but not necessarily so. Even if a crawling job generates data, and a data processing job kicks to process it, and we know that is a linear action-reaction dependency, we can <em>still</em> run data processing regardless, retroactively, as time moves forward and we discover new facts about our data. This includes fixing inconsistencies, taking out stale items, or as a practical example, updating a comment count on items that we’ve discovered earlier.</p> <p><img src="https://rng0.io/blog/analyzing-markets-at-scale-a-crawling-blueprint/6.png" alt="" /></p> <p>To schedule these kind of jobs I recommend Airflow and Jenkins. To make a choice between the two, you’re welcome <a href="https://rng0.io/blog/airflow-the-missing-context/">to read my earlier article about Airflow</a> where I pour some context into the “why Airflow” question.</p> <p>Sticking to Python, we can actually embed Scrapy with our spiders, and a PySpark job into an Airflow DAG.</p> <p>Airflow, unlike Jenkins, will lets us generate crawls dynamically as we see fit. One example is that we can have the same crawl needed to run for different markets, and the markets change with time. Within an Airflow DAG, we generate the current crawl tree based on the current spread of markets; the crawl codebase is always the same but every run tweaks a few settings that turns on or off a bunch of crawl paths.</p> <p>Another nice feature from Airflow over Jenkins is that it has backfilling as a first-class concept. When dealing with crawl data, there’s a ton of mistakes or errornous crawls that we <em>know</em> we want to rerun even shortly after a crawl had started. A great strategy to adopt is one that redoes previous crawls as a safety guard every now and then; Airflow’s backfilling takes away all of the back of envelope calculations to figure out which dates to rerun.</p> <h2 id="data-handling">Data Handling</h2> <p>Let’s imagine we only dealt with files. Each crawl job outputs data feeds with items that it discovered. Other enrichment jobs do the same, dealing with the same structure of data. All jobs have a common way to determine an <code>item_id</code> or to synthesize it from the rest of an item's properties, i.e., as <code>item_url+item_SKU</code>.</p> <p><img src="https://rng0.io/blog/analyzing-markets-at-scale-a-crawling-blueprint/7.png" alt="" /></p> <ol> <li>Crawl by fielding N jobs, we can partition the data by splitting crawling for different sites or different sections of a site, plus jobs for enriching our base crawl data. All jobs output the same item structure, and blanks where missing.</li> <li>Take all new and old data files <code>aggN.json, enrN.json</code> and concatenate, for any number of files that there are, it doesn't really matter. At this stage we <em>know</em> we have duplicates, older runs and new run, and possibly we have duplicates in our own current data run - that's all OK.</li> <li>We now group all data to discover unique items, we merge conflicts when items are the same. One strategy is to take the newer item over an older one when properties conflict, or take a richer property from either copy, and fill blank properties from newest run.</li> <li>We’ve now created a new <em>world view</em> <code>all-agg.json</code> based on <em>all</em> known runs. This view is aggregated and clean, so we can load it onto a relational database for querying, or onto a search engine.</li> </ol> <p>An alternative to step (2) would be to take the <em>latest known</em> worldview, <code>all-aggN.json</code> and concatenate with the new run <code>runN+1.json</code>, resulting in <code>all-aggN+1.json</code>, acknowledging that we're doing a running-average like operation.</p> <p>Trying to reduce this flow to mechanical operations, here’s how it should look like after a few runs.</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain"># &#39;run-N&#39; is a base crawl </span><span class="z-text z-plain"># &#39;e&#39; is an enrichment crawl </span><span class="z-text z-plain"># &#39;wv&#39; is a world view </span><span class="z-text z-plain"> </span><span class="z-text z-plain">run-1, e1 -&gt; wv1 ( = run-1) </span><span class="z-text z-plain">run-2, e2 -&gt; wv2 (= agg(run-2, e2, wv1) </span><span class="z-text z-plain">run-3, e3 -&gt; wv3 (= agg(run-3, e3, wv2)) </span><span class="z-text z-plain">run-4, e4 -&gt; wv4 (= agg(run-4, e4, wv3)) </span><span class="z-text z-plain">... </span></code></pre> <h3 id="using-a-database-vs-files">Using a Database vs Files</h3> <p>Illustrating this process with simple files is both more intuitive and more scalable for when databases can’t hold the dataset we care about. However, when we don’t aim for high scale from the start, we can definitely use a database instead. The big advantage is that we can iterate much faster, have less moving parts and probably pay less if we don’t really have a full big data infrastructure online processing these files.</p> <h3 id="a-files-only">(a) Files only</h3> <p>Data is in a distributed file store and processed with a big data pipeline.</p> <p><strong>Principles</strong></p> <ol> <li>Handle files, since beginning of time, for each revolution of the data model. This lends itself well to a big data — data warehouse settings. Secondly, we <em>always</em> have the raw collected data should we want to go back.</li> <li>Cut chunks of the data flow to atomic processing tasks: deduping, merging and so on.</li> <li>Aggregation using an aggregation model ad-hoc (map/reduce, SAC, SQL over files).</li> <li>Load to one or more data store(s), with an infrastructure and query-optimized data model.</li> </ol> <p><em>Pros</em></p> <ul> <li>Always have raw collected data. With it, we can change the direction of our product in a stateless way, without rummaging through our existing processed datasets trying to figure out what’s good and what’s not. Simply rebuild, on the expense of compute power.</li> <li>Can apply different kinds of fire-power when needed. We can do data processing by employing some batch infrastructure (spark), and data querying and analysis by employing different storage engines or multiple storage engines optimized for retrieval.</li> <li>Better parallelism. We can break creation of data to several processes (e.g. each scraper taking a vertical) and merge files later; as well as perform data enrichment before loading it to a store.</li> <li>Breakable. We can break this proble onto teams or people, assigning tasks to specialty without each stepping on another’s toes.</li> </ul> <p><em>Cons</em></p> <ul> <li>Elaborate to ship data. We need a reliable data shipping process and something to keep track of it.</li> <li>Slower progression. Since we’ve broken up our pipeline we have multiple moving parts and relatively slow communication between those.</li> </ul> <h3 id="b-files-database-hybrid">(b) Files + Database: Hybrid</h3> <p>Using our database as a state book keeper.</p> <p><strong>Principles</strong></p> <ol> <li>Files only exist for the current job; which we get anyway from our crawling infrastructure.</li> <li>Via ETL process, target reach new record in aggregated files against latest world-view in our database to perform merge.</li> <li>Result is a new world view, in our database</li> <li>Rebuilding the world-view is re-applying this process against all files and an empty database serially</li> </ol> <p><em>Pros</em></p> <ul> <li>Simpler. Same process as (a), only that now we use a “scratch book” for book keeping our world view; which means less moving parts (read: files) to aggregate against and coordinate data shipping.</li> <li>We keep scalability for crawling, in case we want to break our crawl data generation to several sub-crawls.</li> </ul> <p><em>Cons</em></p> <ul> <li>Risks of state being corrupted and/or propagating error through the main database.</li> <li>Merging data against our main database will create a stop-the-world like churn every cycle.</li> <li>Scaling up means scaling a database, until we have to dump it.</li> </ul> <h3 id="c-database">(c) Database</h3> <p>Treating our database as an abstract file store.</p> <ol> <li>While crawling, each item is directly inserted into our database (minus trivial deduping) into a current-run “raw” table.</li> <li>A batch SQL job loads “raw” table or multiple “raw” tables and aggregates into a new table, which stands for our current worldview.</li> <li>When our “raw” table(s) represent <em>all</em> data from all previous runs — conceptually similar to (a), when our “raw” table is just the last run against a main table — conceptually similar to (b).</li> </ol> <p><em>Pros</em></p> <ul> <li>No need to deal with files: storage and shipping. Data is shipped via a persistent connection to our database.</li> <li>Saving all data in “raw” form, without processing and indexing, helps with database performance.</li> </ul> <p><em>Cons</em></p> <ul> <li>Single point of failure at worst, dealing with high-availability downtime at best.</li> <li>Waste is generated by constantly pushing raw data and trimming it down by aggregation.</li> </ul> <h3 id="d-database-only-item-loader">(d) Database Only: Item Loader</h3> <ol> <li>Each record discovered is immediately trialed against a database.</li> <li>Datbase helps with deduping.</li> <li>Aggregation is made against the last clean and aggregated state in database.</li> <li>Enrichment or cross-item data processes is done with independent tables and linked via joins.</li> </ol> <p>Pros</p> <ul> <li>Simplest. No file handling, data is shipped easily.</li> <li>No moving parts, an all-in-one, rapid solution.</li> </ul> <p>Cons</p> <ul> <li>Risks of corrupting data propagating errors.</li> <li>High complexity when we want to break data generation (parallel crawls) into multiple stages (each stage competes for database access and data races).</li> <li>Escalates into being a major bottleneck fairly quickly.</li> </ul> <h3 id="aggregating-data-starting-points">Aggregating Data: Starting Points</h3> <p>We’ll take a look at the following dataset, crawled from an imaginary fashion website. This dataset includes two runs (a, b), and part of the items are discovered in both runs, featuring price and rating updates.</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">id,job_id,name,desc,price,reviews,rating,featured </span><span class="z-text z-plain">&quot;1&quot;,a,&quot;black t-shirt&quot;,&quot;a pitch black t-shirt for every occasion&quot;,90.99,303,4.5, </span><span class="z-text z-plain">&quot;1&quot;,b,&quot;black t-shirt&quot;,&quot;a pitch black t-shirt for every occasion&quot;,95.99,303,4.5 </span><span class="z-text z-plain">&quot;1&quot;,a,&quot;black t-shirt&quot;,&quot;a pitch black t-shirt for every occasion&quot;,90.99,308,4.5,1 </span><span class="z-text z-plain">&quot;2&quot;,a,&quot;blue dress&quot;,&quot;a plain blue dress&quot;,110.00,2,2.0,0 </span><span class="z-text z-plain">&quot;2&quot;,a,&quot;blue dress&quot;,&quot;a plain blue dress&quot;,110.00,2,2.0,1 </span><span class="z-text z-plain">&quot;2&quot;,a,&quot;blue dress&quot;,&quot;a plain blue dress&quot;,110.00,2,2.0,1 </span><span class="z-text z-plain">&quot;2&quot;,b,&quot;blue dress&quot;,&quot;a plain blue dress&quot;,110.00,2,2.0,1 </span></code></pre> <p>We need to take this data set, and aggregate it. Here are a few notes about this aggregation.</p> <ol> <li>First we want to build a record set that forms the baseline of all our items, an “epoch” of our worldview. We can take these from our first job (job id=a).</li> <li>Next, perform diff tracking by identifying items that <em>changed</em> through all successive jobs (job id=b, etc.), and only for those items add records that describe an updated item. Items that are completely identical <em>between</em> runs are dropped.</li> <li>Diff tracking logic can change from application to application. For this instance, we say that an item can still be the same if it has a different description (we don’t care we just take the first one we see), but we observe a change if its price changed.</li> </ol> <p>Here’s how it looks like using <a href="https://pandas.pydata.org">pandas</a>, a data science oriented toolkit:</p> <p>And <a href="https://petl.readthedocs.io">petl</a>, a slightly more ETL oriented toolkit:</p> <p>For this last example, I’ve used a different dataset that I pulled from production. Avoiding the annoyance of including this data set here, the principles are the same minus the field names.</p> <p>Lastly, if you want to strictly deal with putting items on a database, I’ve encoded a flexible set of conflict resolution that you can use for diff tracking logic into <a href="https://github.com/jondot/pgpipeline">pgpipeline</a>, a Scrapy plugin that hooks directly into your spider pipeline and with it, you can describe how to add, index, merge and track items.</p> <h2 id="summary">Summary</h2> <p>The hardest task for building this kind of product is making the right choices for each component and building something that can start small, iterate fast, and scale up with little to no effort.</p> <p>The architecture and technical choices presented here started small and scaled up without requiring re-engineering. It’s delivering a market analysis product that works at scale, covering stores with tens of thousands of products.</p> <p>There’s a more detail to share, but I’ve been painfully careful to not have a narrow view and describe this one specific product but a family of products fitting this domain, so that you can observe the pattern, context, and decisions that are needed to succeed.</p> Airflow: The Missing Context Thu, 12 Oct 2017 00:00:00 +0000 Unknown https://rng0.io/blog/airflow-the-missing-context/ https://rng0.io/blog/airflow-the-missing-context/ <p>There are a number of ways to do data workflows:</p> <ul> <li>Custom scripts</li> <li>Cron, running each discrete task in a workflow based on timing and run-time of each task</li> <li><a href="https://jenkins.io/">Jenkins</a>, modeling each task as a job, and using job dependencies</li> <li><a href="http://oozie.apache.org/">Oozie</a>, <a href="https://github.com/spotify/luigi">Luigi</a>, and other tailor made workflow management systems born into the Hadoop ecosystem</li> <li><a href="https://airflow.incubator.apache.org">Airflow</a></li> </ul> <p>We’re going to take a look at some of these approaches and end up with a discussion of why <a href="https://airflow.incubator.apache.org/">Airflow</a> is the better choice.</p> <h2 id="our-use-case">Our use case</h2> <p>Let’s build a price tracking platform, which tracks a popular ecommerce website, let’s say Ebay’s <a href="https://www.ebay.com/rpp/fashion-main">fashion section</a>, so that we can cross-reference products there with other fashion sites like Zappos and Zara. Of course, we’re talking “in theory” here, because crawling websites that don’t allow it, is against their TOS. Moving on, here are the high level steps.</p> <p><img src="https://rng0.io/blog/airflow-the-missing-context/1.png" alt="" /></p> <p>Deep crawl over one or more stores. In this step we’re crawling one more more websites. We can crawl and dump everything we see, or scrape most of the data we need, dumping a semi-structured and semi-clean data to disk.</p> <p><img src="https://rng0.io/blog/airflow-the-missing-context/2.png" alt="" /></p> <p>Cleansing, normalization and structured data extraction. In this step we’ll get rid of blanks, nulls, corrupted, and redundant data points. We’ll try our best to normalize URLs, domains, and the likes to their canonical form; potentially discovering duplicate records.</p> <p><img src="https://rng0.io/blog/airflow-the-missing-context/3.png" alt="" /></p> <p>Enrichment. Although we’re getting data from the source, there’s still auxiliary data such as currency rates, inventory, blacklists, and so on that we need to use for enriching our existing data. These kind of data can be had by running more crawls, fetching data from APIs, or joining against a database.</p> <p><img src="https://rng0.io/blog/airflow-the-missing-context/4.png" alt="" /></p> <p>Merge (match enrichment to items). Previous steps left us with data points that are clean and that we can join; perhaps we need to compute a currency on an item, based on an exchange rate we’ve fetched, or we need to fill-in a missing field in some items, from another, auxiliary sub-crawl that we ran as part of the enrichment step.</p> <p><img src="https://rng0.io/blog/airflow-the-missing-context/5.png" alt="" /></p> <p>Aggregation and trend tracking. In this step we care about deduplicating data, and figuring out the identity of items, so that for every item, we can cross reference these with previous runs and see if in that item really changed. If it did, we’d output a change record, which allows us to model change-tracking, and then detecting trends in each item individually based on the rate of change and properties of that change.</p> <p>As a workflow, we can execute these steps linearly as listed, but because a good part of the steps are not dependent, they are a graph that lends itself to optimization. For example, we can run the main crawl <em>while</em> running enrichment crawls, and continue when both finished.</p> <h2 id="workflows-with-make">Workflows with Make</h2> <p><img src="https://rng0.io/blog/airflow-the-missing-context/6.png" alt="" /></p> <p>Assuming we want to track fashion items on ebay, here’s a clunky workflow with <code>make</code>.</p> <pre data-lang="makefile" class="language-makefile z-code"><code class="language-makefile" data-lang="makefile"><span class="z-source z-makefile"><span class="z-meta z-function z-makefile"><span class="z-entity z-name z-function z-makefile">crawl-main</span></span><span class="z-keyword z-operator z-assignment z-makefile">:</span> <span class="z-meta z-function z-arguments z-makefile"></span><span class="z-meta z-function z-body z-makefile"></span></span><span class="z-source z-makefile"><span class="z-meta z-function z-body z-makefile"></span><span class="z-meta z-function z-body z-makefile"> <span class="z-source z-shell"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">scrapy</span></span><span class="z-meta z-function-call z-arguments z-shell"> crawl ebay_wear</span></span> </span></span><span class="z-source z-makefile"><span class="z-meta z-function z-body z-makefile"></span><span class="z-meta z-function z-makefile"><span class="z-entity z-name z-function z-makefile">crawl-enrich</span></span><span class="z-keyword z-operator z-assignment z-makefile">:</span> <span class="z-meta z-function z-arguments z-makefile"><span class="z-string z-unquoted z-makefile">crawl-main</span></span><span class="z-meta z-function z-body z-makefile"> </span></span><span class="z-source z-makefile"><span class="z-meta z-function z-body z-makefile"></span><span class="z-meta z-function z-body z-makefile"> <span class="z-source z-shell"><span class="z-variable z-other z-readwrite z-assignment z-shell">RECENCY_ONLY</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell">1</span> <span class="z-variable z-other z-readwrite z-assignment z-shell">IN_7</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell">1</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">scrapy</span></span><span class="z-meta z-function-call z-arguments z-shell"> crawl zara</span></span> </span></span><span class="z-source z-makefile"><span class="z-meta z-function z-body z-makefile"> <span class="z-source z-shell"><span class="z-variable z-other z-readwrite z-assignment z-shell">RECENCY_ONLY</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell">1</span> <span class="z-variable z-other z-readwrite z-assignment z-shell">IN_30</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell">1</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">scrapy</span></span><span class="z-meta z-function-call z-arguments z-shell"> crawl h_n_m</span></span> </span></span><span class="z-source z-makefile"><span class="z-meta z-function z-body z-makefile"> <span class="z-source z-shell"><span class="z-variable z-other z-readwrite z-assignment z-shell">RECENCY_ONLY</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell">1</span> <span class="z-variable z-other z-readwrite z-assignment z-shell">IN_90</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell">1</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">scrapy</span></span><span class="z-meta z-function-call z-arguments z-shell"> crawl zappos</span></span> </span></span><span class="z-source z-makefile"><span class="z-meta z-function z-body z-makefile"></span><span class="z-meta z-function z-makefile"><span class="z-entity z-name z-function z-makefile">agg-last</span></span><span class="z-keyword z-operator z-assignment z-makefile">:</span> <span class="z-meta z-function z-arguments z-makefile"><span class="z-string z-unquoted z-makefile">crawl-enrich</span></span><span class="z-meta z-function z-body z-makefile"> </span></span><span class="z-source z-makefile"><span class="z-meta z-function z-body z-makefile"></span><span class="z-meta z-function z-body z-makefile"> <span class="z-source z-shell"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">python</span></span><span class="z-meta z-function-call z-arguments z-shell"> process_results.py</span></span> </span></span></code></pre> <p>This (dangerously) works for the happy path (and sad developer) — there’s no failure recovery and no operational story at all and maintainbility is horrible. It’s run-and-pray and then troubleshoot by hand.</p> <p>Some tasks are verbatim in the <code>Makefile</code> such as the <code>crawl-</code> tasks, and knobs appear as environment variables. Other tasks are more cryptic, like <code>agg-last</code>, where we shell out to Python because what ever we tried doing is too complex to remain in <code>make</code> realm.</p> <p>And we run this by saying:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-variable z-other z-readwrite z-assignment z-shell">JOB_ID</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell">run-11</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">make</span></span><span class="z-meta z-function-call z-arguments z-shell"> agg-last</span> </span></code></pre> <p>When we run <code>make</code> in this way we synthesize a concept of a unit of work, so that all processes spawned are aware of it and build artifacts under it. By saying <code>JOB_ID=run-11</code> each one of the processes <em>must</em> now implicitly be aware of it and generate an output under an agreed upon <code>run-11</code> folder.</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">/out </span><span class="z-text z-plain"> /run-10 </span><span class="z-text z-plain"> ebay_wear.csv </span><span class="z-text z-plain"> zara.csv </span><span class="z-text z-plain"> h_n_m.csv </span><span class="z-text z-plain"> zappos.csv </span><span class="z-text z-plain"> merged.csv </span><span class="z-text z-plain"> agged.csv </span><span class="z-text z-plain"> /run-11 </span><span class="z-text z-plain"> ebay_wear.csv </span><span class="z-text z-plain"> zara.csv </span><span class="z-text z-plain"> h_n_m.csv </span><span class="z-text z-plain"> zappos.csv </span><span class="z-text z-plain"> merged.csv </span><span class="z-text z-plain"> agged.csv </span></code></pre> <p>But what was the core idea here going for <code>make</code> in the first place? <code>make</code> <em>does</em> know how to execute a model of a tree of dependencies, and it's already proven for highly complex builds with tons of dependencies, and heck, we can also play with making sentinel <code>crawl-main</code>, <code>crawl-enrich</code>, and <code>agg-last</code> files to have <code>make</code> skip tasks that have we ran successfully.</p> <p>Bottom line, the intuition might be there, but the impedence mismatch between a build system and a data workflow management system has to result with a clunky solution sporting bad ergonomics that is hard to maintain and troubleshoot.</p> <h2 id="workflows-with-code">Workflows with Code</h2> <p>Here’s how some of the workflow should look like in pure code. The main takeaways here are that we generate a job ID and around that a working directory for that job manually. We then have no choice but to build our own operators such as <code>crawl</code> so that the workflow is <em>somewhat</em> readable. There's still plenty that is repetitive and subpar.</p> <p>This marks a move away from our declarative attempt with <code>make</code> and into a programmatic attempt with a Python script. Nothing special about Python here, aside of the fact that both crawling and data wrangling frameworks used later are in Python as well, so that it makes sense to keep at it.</p> <h2 id="workflows-with-jenkins">Workflows with Jenkins</h2> <p>Building workflows with Jenkins <a href="https://wiki.jenkins.io/display/JENKINS/Dependency+Graph+View+Plugin">can be great</a>, and for a while I’ve ignored Luigi, Oozie and Azkaban in favor of using Jenkins. It can offer everything that Airflow provides and more in form of Jenkins plugins and its ecosystem and before you’re done with modeling data jobs, you can integrate your existing CI jobs for the code that wrangles your data directly with your data pipeline.</p> <p>You get dependency modeling between jobs, job ID and isolation, artifacts, history, gantt, visualizations <a href="https://plugins.jenkins.io/">and more</a>, and lastly a distributed model for workers that’s battle proven over <a href="https://plugins.jenkins.io/mesos">Mesos</a>, <a href="https://plugins.jenkins.io/kubernetes">Kubernetes</a> or what-have-you, including its built-in robust scheduling and triggering mechanisms.</p> <p>If your data pipeline work doesn’t require dynamic workflows, complex workflows, and extreme flexibility, then by all means, there’s no reason why you can’t model your data pipeline workflows in Jenkins. It’ll be simpler and closer to what your probably already know.</p> <p>Otherwise, it’d be tedious, and it won’t be a platform you can meaningfully evolve and remain within your data realm. For example, while you could build plugins for customizing your workflow platform (read: Jenkins), it’d be a horrible experience to build, and probably a nightmare getting a grasp of Jenkins’ internal APIs, compared to Airflow’s small API surface area, and its ‘add a script and import’ ergonomics.</p> <h2 id="workflows-with-airflow">Workflows with Airflow</h2> <p><img src="https://rng0.io/blog/airflow-the-missing-context/7.png" alt="" /></p> <p>Here’s a minimal DAG with Airflow with some naive configuration to keep this example readable.</p> <p>I like to abstract operator creation, as it ultimately makes a more readable code block and allows for extra configuration to generate dynamic tasks, so here we have <code>crawl</code>, <code>combine</code>, <code>agg</code>, <code>show</code> and all can take parameters.</p> <p>Here’s the <code>crawl</code> function.</p> <p>The oddly looking <code>{{{{ds}}}}</code> bit is what makes our job ID. This is a special template variable that Airflow injects for us for free - this <code>bash_command</code> parameter is actually a string template, passed into Airflow, rendered, and <em>then</em> executed as a Bash command. Because Airflow makes time a first-class citizen, you can look at plenty more of those special parameters <a href="https://airflow.incubator.apache.org/code.html#default-variables">here</a>.</p> <p>Hidden from our view is the metadata store and the scheduler and executor. The metadata store is a database that stores records of previous task runs and DAG runs, which makes task not repeat if they ran, and lets us retry, force-run tasks and DAGs and more. The <a href="https://airflow.incubator.apache.org/code.html#executors">executor</a> can be a local executor using local processes, or a distributed one using a job queue or any other scheduler that we can build.</p> <p>DAG authoring is ignorant of the operational aspects of itself: state and execution. This means that if we wanted to let data scientists write DAGs, devops maintain the database and data engineers work on the execution engine and DAG building blocks ( <a href="https://airflow.incubator.apache.org/code.html#operators">operators</a>, <a href="https://airflow.incubator.apache.org/code.html#basesensoroperator">sensors</a>) — we could.</p> <h3 id="the-good-parts">The Good Parts</h3> <h3 id="dags">DAGs</h3> <p>Although Airflow uses the term “DAG” everywhere, it doesn’t make that much difference. The term is making its odd hype comeback, I guess. You build and use DAGs every day when you’re sorting out your builds and their dependencies. DAGs are everywhere, regardless of a framework or dealing with data. But if you didn’t happen to go over graph theory in university then it’ll probably make a difference in your thinking.</p> <p>That said, plenty of your workflow tasks will be linear at first, and those that don’t will require plenty of fiddling with, and that will all happen as you go. Most often a job will start as one task, break into a linear series of jobs, perhaps running on different capacity machines, and then you’d optimize it for running bits in parallel, converge, and have steps communicate via a distributed file system.</p> <h3 id="dag-control">DAG Control</h3> <p>The ability to visualize a DAG in more than one way — graph, tree, Gantt, and more importantly — to interact with such a DAG by selecting a node and forcing repeat, re-running a failed node, force running an entire DAG, looking at a node’s progress as part of a DAG run and its logs do make a difference, and stand out from any other system that uses “DAG” to promote itself.</p> <h3 id="ergonomics">Ergonomics</h3> <p>The API feels programmatic. Although it isn’t the most concise one, generating dynamic workflows and building new APIs that use it almost asks to happen as you start working with Airflow. Operators, Sensors and Hooks are simple concepts and yet carry a lot of weight so that when you subclass you get a ton of functionality.</p> <p>Dynamic workflows are the killer feature. It’s nice to spin a for-loop that generates an array of parallel tasks for you and saves you some typing, but also think of Airflow and its malleability as a library you can use in your own projects.</p> <p>With Airflow, you can have self-assembling workflows, dynamic and parameter-bound, and you can build one of those cool data shipping startups that hose data from one place to another, effectively building a multi-tenant workflow system and executor as-a-service like AWS data pipelines. For example, you can generate DAGs based on configuration files or database rows where each belongs to a different client.</p> <h3 id="time">Time</h3> <p>Time is a first-class citizen. But more abstractly, timing is versioning; if you’re dealing with software then it’s probable that you have some scheme of versioning built-in, from which you do delivery through periodical iterations, branches, patch and hotfix strategies, and perhaps a feature matrix.</p> <p>When you’re dealing with data ingress, and organic traffic, then its most probable a time window is your de-facto version to use. For example you probably synthesized something like <code>dt=20170908</code> as a folder name, file name, and log tag, to model a "version" of the current data iteration.</p> <p>The impedence mismatch between time represeting a temporal thing, and version representing a discrete incremental thing, will vary between not-quite-a-workflow-management-system like Jenkins, to workflow management systems like Luigi, to data workflow management systems like Airflow.</p> <p>In Airflow, there’s a strong understading of time being the natural cadence for data moving forward. For example, a time stamp should be your “job ID”. Once we commit to specializing the versioning concept over data as a time window, then we can talk about time-specific operations such as “backfilling” based on start and end date, time intervals, scheduling, and so on. In other systems where time plays an equal role as any other concept, you’d have to manually make that work.</p> <h3 id="growth">Growth</h3> <p>Hooks, Operators and Sensors are basic building blocks which Airflow relies on expanding for better growth and adoption as a project. I feel that there’s almost a bet Airflow will prosper as a project iff people build more of these, which is not too hard to build.</p> <p>Actually running a task using an Airflow worker’s cpu cycles vs an Airflow worker <em>triggering</em> a task in a remote, more powerful cluster allow for simplicity. More people can use the first case, and even a hybrid model using inline Spark (pyspark, at least) code, in the DAG file, which is easy and thereby attractive, and lastly go with the latter case where an Airflow task merely triggers a job in a more resourceful machine.</p> <h3 id="the-bad-parts">The Bad Parts</h3> <p>Airflow has some bad parts, but to take some care away I think whoever uses Airflow is still in the initial circle of early adopters; and this is a bleeding edge product, so expect to bleed.</p> <h3 id="logs">Logs</h3> <p>Airflow is still clunky in some parts. There’s implicitness going on with automatic loading of DAGs, components that are separate from each other, which is a good thing; web, scheduler, executor, database, and more. All this means that when something goes wrong we have plenty of sifting through these components to understand what went wrong and then how to fix it; locating the right task and the right log is time consuming.</p> <p>The exciting part is that this feels like a low-hanging-fruit; one idea is to show errors on the DAG visualization itself in a form of a badge with error count, and clicking-through will get you to the exact point in the logs.</p> <h3 id="context">Context</h3> <p>Although the documentation <em>feels</em> comprehensive, its not. There’s a lot of context missed (hopefully this article closes some gap there), and it feels that everything is there, in form of examples and building blocks, but no one took the time yet to do the appropriate write-ups.</p> <p>It feels that with some more time it’ll be more complete; but on the other side the Airflow adoption train is moving ahead, and I keep spotting confused souls that I had to give this context to.</p> <h3 id="bugs">Bugs</h3> <p>That Airflow makes such a good fit for data workflows, and makes a no-brainer decision to adopt, obscures the fact that this is still a young product that tackles a complex topic. There’s bugs and plenty hidden knobs to bump into (as of v1.8.1), even when you’re just starting up. For the hidden knobs I recommend <a href="https://airflow.incubator.apache.org/code.html">giving the API reference</a> a thorough read, and for the bugs, make sure to get the latest version and be ready to go to Github issues every time you feel something is odd.</p> <p>Lastly, don’t forget that because of the couple previous bad parts; logs and context — you’d be out ways to get a bearing for what’s wrong when something goes wrong — so for that, reading <a href="https://github.com/apache/incubator-airflow">Airflow’s code</a> is a good way to ground yourself with some more intuition.</p> <h3 id="conclusion">Conclusion</h3> <p>This wasn’t an Airflow tutorial or a setup guide, there’s <a href="https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links">plenty of those already</a>. It’s an attempt to hit the gray area where word-of-mouth is, and give you some good context for what to expect and how to prepare when you decide to adopt Airflow.</p> <p>Airflow makes for a good match for <em>data specific</em> workflows because it understands time, its use case and how we build workflows, and makes a great tool built for data engineering.</p> A Story of a Fat Go Binary Fri, 17 Feb 2017 00:00:00 +0000 Unknown https://rng0.io/blog/story-of-a-fat-go-binary/ https://rng0.io/blog/story-of-a-fat-go-binary/ <p><em>Update: With Go 1.11 and the new module system, we needed a more sophisticated tool — announcing</em> <code>goweight</code> <em>—</em> <a href="https://github.com/jondot/goweight"><em>check out the repo</em></a> _._Small deliverables are important for CD scenarios and low-resource applications. In this article we’ll focus on building an agent that should run on different kinds of low resource devices — RAM, CPU, and actually — we can’t predict how low.</p> <p>Go binaries are small and self-contained: when you build a Go program the resulting single binary is all there is. This is in contrast to platforms like Java, Node.js, Ruby, and Python where you may feel <em>your</em> code is small, but then there’s a mountain of dependencies behind it that you need to pack for a self-contained deliverable.</p> <p>While having a self-contained binary is an important convenience, Go doesn’t have a built-in way for conveniently reasoning about the size of dependencies, so that one can make informed decisions about including them or not. To do this, we’ll introduce <code>gofat</code>, a tool which lets you break down dependency sizes of a Go project.</p> <p>But first, let’s start our story.</p> <h2 id="building-an-iot-agent">Building an IoT Agent</h2> <p>Let’s breeze through a story that shows how to reason about and build a service — an IoT agent — that we’ll deploy to a modest hardware device somewhere across the globe. We will highlight the architecture of such an agent from an operations point of view.</p> <p><em>You can grab the sample code</em> <a href="https://github.com/jondot/fattyproject"><em>here</em></a> <em>.</em></p> <p>First, we want good CLI ergonomics, so we’ll use <a href="https://github.com/alecthomas/kingpin">kingpin</a>, which is a POSIX compliant CLI flags/opts library; it’s been such a great library that I’ve been carrying it over a lot of my projects by default.</p> <p>In fact, I’ll use my <a href="https://github.com/jondot/go-cli-starter">go-cli-starter</a> project that includes it:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> git clone https://github.com/jondot/go-cli-starter fattyproject</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Cloning</span></span><span class="z-meta z-function-call z-arguments z-shell"> into <span class="z-string z-quoted z-single z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&#39;</span>fattyproject<span class="z-punctuation z-definition z-string z-end z-shell">&#39;</span></span>...</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">remote:</span></span><span class="z-meta z-function-call z-arguments z-shell"> Counting objects: 55, done.</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">remote:</span></span><span class="z-meta z-function-call z-arguments z-shell"> Total 55 (delta 0</span><span class="z-meta z-function-call z-shell"></span>)<span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">,</span></span><span class="z-meta z-function-call z-arguments z-shell"> reused 0 (delta 0</span><span class="z-meta z-function-call z-shell"></span>)<span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">,</span></span><span class="z-meta z-function-call z-arguments z-shell"> pack-reused 55</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">Unpacking</span></span><span class="z-meta z-function-call z-arguments z-shell"> objects: 100<span class="z-meta z-group z-expansion z-job z-shell"><span class="z-punctuation z-definition z-variable z-job z-shell">%</span></span> (55/55</span><span class="z-meta z-function-call z-shell"></span>)<span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">,</span></span><span class="z-meta z-function-call z-arguments z-shell"> done.</span> </span></code></pre> <p>As an agent, we want to stay always up. We’ll do that with a dummy <em>forever-loop</em> that’s doing nonsense, generally, just for the sake of this exercise.</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-keyword z-control z-go">for</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">f</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-function z-go">NewFarble</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-keyword z-operator z-go">&amp;</span><span class="z-variable z-other z-go">Counter</span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span><span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">f</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Bumple</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">time</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Sleep</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">time</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">Second</span> <span class="z-keyword z-operator z-go">*</span> <span class="z-constant z-numeric z-integer z-decimal z-go">1</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <p>Long running processes accumulate cruft — small traces of memory leaks, forgotten open file descriptors — even the smallest of leaks become huge when we’re talking a year-long up times.</p> <p>You’d be happy to know that Go has a built-in metrics and health facility called <code>expvars</code>. This will be perfect to expose the agent's inner-workings: the idea is that since an agent is long-running, some times we'll have forensics sessions where we'll want to understand how's an agent doing - CPU, GC cycles and so on and <code>expvars</code> can do it for us, and also, <a href="https://github.com/divan/expvarmon">expvarmon</a> is pretty cool to use for that.</p> <p>To use <code>expvars</code> we need a magic import. Magic because it will find and add an endpoint to an existing HTTP listener; so we need to have an HTTP endpoint up and running. We'll take that from <code>net/http</code>.</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-keyword z-other z-import z-go">import</span> <span class="z-punctuation z-section z-parens z-begin z-go">(</span> </span><span class="z-source z-go"> <span class="z-variable z-language z-blank z-go">_</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>expvar<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>net/http<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span><span class="z-source z-go"> </span><span class="z-source z-go"> <span class="z-punctuation z-separator z-go">:</span> </span><span class="z-source z-go"> <span class="z-punctuation z-separator z-go">:</span> </span><span class="z-source z-go"> </span><span class="z-source z-go"><span class="z-keyword z-control z-go">go</span> <span class="z-storage z-type z-keyword z-function z-go">func</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">ListenAndServe</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>:5160<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-constant z-language z-go">nil</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></code></pre> <p>Since we’re becoming a sophisticated service, we might as well add a leveled logging facility to differentiate between normal activity, errors, and warnings. A good one is <code>zap</code>, by Uber.</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-keyword z-other z-import z-go">import</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span> </span><span class="z-source z-go"> <span class="z-punctuation z-separator z-go">:</span> </span><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>go.uber.org/zap<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span><span class="z-source z-go"> <span class="z-punctuation z-separator z-go">:</span> </span><span class="z-source z-go"> </span><span class="z-source z-go"><span class="z-variable z-declaration z-go">logger</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-language z-blank z-go">_</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">zap</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">NewProduction</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span><span class="z-source z-go"><span class="z-variable z-other z-go">logger</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Info</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>OK<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">zap</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Int</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>ip<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">ip</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></code></pre> <p>A service that’s always on, running in a remote device you can’t control — and most probably can’t update — is very rigid. It makes sense to bake in flexibility of some sort. One trick is to have it run custom commands, scripts, or basically — have it change its behavior without redeploying or restarting.</p> <p>We’ll add a facility to run an arbitrary remote script. Although borderline suspicious, if this is <em>your</em> agent or service, then you can prepare an embedded runtime sandbox to run code which makes it OK. Two such runtimes that are popular to embed are Javascript and Lua.</p> <p>We’ll use an embedded Javascript engine called <a href="https://github.com/robertkrimen/otto">otto</a>.</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-keyword z-other z-import z-go">import</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span> </span><span class="z-source z-go"> <span class="z-punctuation z-separator z-go">:</span> </span><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>github.com/robertkrimen/otto<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span><span class="z-source z-go"> <span class="z-punctuation z-separator z-go">:</span> </span><span class="z-source z-go"> </span><span class="z-source z-go"><span class="z-keyword z-control z-go">for</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-separator z-go">:</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">vm</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Run</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-other z-go"><span class="z-punctuation z-definition z-string z-begin z-go">`</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-string z-quoted z-other z-go"> abc = 2 + 2; </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-string z-quoted z-other z-go"> console.log(&quot;\nThe value of abc is &quot; + abc); // 4 </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-string z-quoted z-other z-go"> <span class="z-punctuation z-definition z-string z-end z-go">`</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-separator z-go">:</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <p>Now, as long as we fetch the content that we stick in <code>Run</code> from a remote endpoint - we've got a sophisticated, self-updating, IoT agent!</p> <h3 id="understanding-go-binary-dependencies">Understanding Go Binary Dependencies</h3> <p>Let’s look at what we’ve got so far.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> ls<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>lha</span> fattyproject</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">...</span></span><span class="z-meta z-function-call z-arguments z-shell"> 13M ... fattyproject<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span></span> </span></code></pre> <p>Having that the dependencies we added are reasonable, we have caused our binary to creep up to 12MB in size. I still see this as a <em>tiny</em> binary in comparison to other languages and platforms; but from an modest IoT hardware point of view, anything we can give up for a simpler or smaller overhead and size can be useful.</p> <p>Our first task is to understand how did the dependencies in this binary add up.</p> <p>Let’s break down a well known binary first. <a href="http://www.graphicsmagick.org/">GraphicsMagick</a> is a modern variation on the well known ImageMagick image processing system, and you probably already have that installed. If not, it’s a <code>brew install graphicsmagick</code> away on OSX.</p> <p>Then, <code>otool</code> is an alternative for <a href="http://man7.org/linux/man-pages/man1/ldd.1.html">ldd</a> on OSX. With it, we can break down a binary and see what kind of libraries it's linked to.</p> <p>We can survey a dependency size by picking it up from the listing:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> ls<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>lha</span> /usr/l/.../-0_2/lib/libMagickCore-6.Q16.2.dylib</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">...</span></span><span class="z-meta z-function-call z-arguments z-shell"> 1.7M ... /usr/.../libMagickCore-6.Q16.2.dylib</span> </span></code></pre> <p>Can we build a good mental map of <em>any</em> binary in this way? (apparently, the answer is “No”.)</p> <p>Go links its dependencies statically by default. This has the benefit of having one simple and portable deliverable — the binary itself. This <em>also</em> means that <code>otool</code> or any such binary-first tool would be useless.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> cat main.go</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">package</span></span><span class="z-meta z-function-call z-arguments z-shell"> main</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">func</span></span><span class="z-meta z-function-call z-arguments z-shell"> main(</span><span class="z-meta z-function-call z-shell"></span>) <span class="z-punctuation z-definition z-compound z-braces z-begin z-shell">{</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">print</span></span><span class="z-meta z-function-call z-arguments z-shell">(<span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>hello<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span></span><span class="z-meta z-function-call z-shell"></span>) </span><span class="z-source z-shell z-bash"><span class="z-punctuation z-definition z-compound z-braces z-end z-shell">}</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> go build</span> <span class="z-keyword z-operator z-logical z-and z-shell">&amp;&amp;</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">otool</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>L</span> main</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">main:</span></span> </span></code></pre> <p>To try and still break down a Go binary to its dependencies, we must use a Go-enlightened tool that can understand the Go binary format. Let’s find one.</p> <p>To get a dump of the available tools, use <code>go tool</code>:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> go tool</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">addr2line</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">api</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">asm</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">cgo</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">compile</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">cover</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">dist</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">doc</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">fix</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">link</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">nm</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">objdump</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">pack</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">pprof</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">trace</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">vet</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">yacc</span></span> </span></code></pre> <p>You can dive right into the <a href="https://golang.org/src/cmd/">source listing of these</a>, and if we look at the <code>nm</code> tool, for example, we can view its package documentation in <a href="https://golang.org/src/cmd/nm/doc.go">src/cmd/nm/doc.go</a>.</p> <p>I pointed out <code>nm</code> intentionally. As it happens, this tool is exotically close to what we're trying to do, but not close enough. It may list symbols and object sizes, but none of that makes sense if we're trying to make up the dependencies of a binary in the larger sense.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> go tool nm<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>sort</span> size<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>size</span> fattyproject</span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">head</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>n</span> 20</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">5ee8a0</span></span><span class="z-meta z-function-call z-arguments z-shell"> 1960408 R runtime.eitablink</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">5ee8a0</span></span><span class="z-meta z-function-call z-arguments z-shell"> 1960408 R runtime.symtab</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">5ee8a0</span></span><span class="z-meta z-function-call z-arguments z-shell"> 1960408 R runtime.pclntab</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">5ee8a0</span></span><span class="z-meta z-function-call z-arguments z-shell"> 1960408 R runtime.esymtab</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">4421e0</span></span><span class="z-meta z-function-call z-arguments z-shell"> 1011800 R type.<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span></span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">4421e0</span></span><span class="z-meta z-function-call z-arguments z-shell"> 1011800 R runtime.types</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">4421e0</span></span><span class="z-meta z-function-call z-arguments z-shell"> 1011800 R runtime.rodata</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">551a80</span></span><span class="z-meta z-function-call z-arguments z-shell"> 543204 R go.func.<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span></span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">551a80</span></span><span class="z-meta z-function-call z-arguments z-shell"> 543204 R go.string.hdr.<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span></span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">12d160</span></span><span class="z-meta z-function-call z-arguments z-shell"> 246512 T github.com/robertkrimen/otto._newContext</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">539238</span></span><span class="z-meta z-function-call z-arguments z-shell"> 100424 R go.string.<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span></span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">804760</span></span><span class="z-meta z-function-call z-arguments z-shell"> 65712 B runtime.trace</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">cd1e0</span></span><span class="z-meta z-function-call z-arguments z-shell"> 23072 T net/http.init</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">5e3b80</span></span><span class="z-meta z-function-call z-arguments z-shell"> 21766 R runtime.findfunctab</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">1ae1a0</span></span><span class="z-meta z-function-call z-arguments z-shell"> 18720 T go.uber.org/zap.Any</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">301510</span></span><span class="z-meta z-function-call z-arguments z-shell"> 18208 T unicode.init</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">5e9088</span></span><span class="z-meta z-function-call z-arguments z-shell"> 17924 R runtime.typelink</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">3b7fe0</span></span><span class="z-meta z-function-call z-arguments z-shell"> 16160 T crypto/sha512.block</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">8008a0</span></span><span class="z-meta z-function-call z-arguments z-shell"> 16064 B runtime.semtable</span> </span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">3f6d60</span></span><span class="z-meta z-function-call z-arguments z-shell"> 14640 T crypto/sha256.block</span> </span></code></pre> <p>The numbers above may be accurate for dependencies (size is in the second column), for example <code>_newContext</code> from the <code>otto</code> package, but the math might be a bit involved or missing for the general sense.</p> <h2 id="gofat">Gofat</h2> <p>There’s one last trick that will work. When you compile your Go binary, Go will generate interim binaries for each dependency, before statically linking these all up into the one binary you get in the end.</p> <p>Introducing <code>gofat</code> — a shell script that’s a mix of Go and some Unix tools that analyzes a Go binary dependencies sizes:</p> <p>If you’re in a hurry, just copy or download the above shell script and set it to be executable ( <code>chmod +x</code>). Then, run it in your project’s directory with no arguments in order to get that project’s dependency breakdown.Let’s pick this command apart.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-support z-function z-eval z-shell">eval</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-meta z-group z-expansion z-command z-backticks z-shell"><span class="z-punctuation z-section z-group z-begin z-shell">`</span><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">go</span></span><span class="z-meta z-function-call z-arguments z-shell"> build<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>work</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>a</span> <span class="z-constant z-numeric z-integer z-decimal z-file-descriptor z-shell">2</span><span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;&amp;</span><span class="z-constant z-numeric z-integer z-decimal z-file-descriptor z-shell">1</span></span><span class="z-punctuation z-section z-group z-end z-shell">`</span></span></span> </span></code></pre> <p>Using the <code>-a</code> flag, we're telling Go to ignore any cache and build a project from scratch— this will force a build of all dependencies. Using <code>-work</code> outputs a working dir environment variable export pragma, so we eval that (thanks Go team!).</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">find</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">WORK</span></span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>type</span> f<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>name</span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>*.a<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span></span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">xargs</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>I</span><span class="z-meta z-group z-expansion z-brace z-shell"><span class="z-punctuation z-section z-expansion z-brace z-begin z-shell">{</span><span class="z-punctuation z-section z-expansion z-brace z-end z-shell">}</span></span> du<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>hxs</span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>{}<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span></span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">gsort</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>rh</span></span> </span></code></pre> <p>Having the WORK environment variable populated and pointing into our working build directory, we now look for all <code>*.a</code> files, which represent the compiled form of our dependencies with the <code>find</code> tool.</p> <p>We then feed all lines, which are file locations, into <code>xargs</code> — which in turn is a utility that lets you run a command on each of the piped lines — in our case, into <code>du</code> that takes a size of a file.</p> <p>Finally, we use gsort (the GNU version of sort) to perform a reverse sort of the sizes.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sed</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>e</span> s:<span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">WORK</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span>/::g</span> </span></code></pre> <p>We strip out a prefix of the WORK folder from everything we’ve got and we display a nice and clean dependency string.Let’s get to the fun part of seeing what’s taking up those 12MB in our binary!</p> <h2 id="trimming-down-the-fat">Trimming Down the Fat</h2> <p>Let’s run <code>gofat</code> for the first time on our mock IoT agent project. Here’s the output:</p> <p>If you’re trying this yourself, you’ll notice build times are considerably longer with <code>gofat</code>. This is because we're running a build in <code>-a</code> mode, which means rebuild everything.</p> <p>Now that we know how much space each dependency is taking, let’s roll up our sleeves and make observations — and decisions.</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">1.8M net/http.a </span></code></pre> <p>Doing anything related to HTTP handling costs 1.8MB. We can probably drop this, and not use <code>expvars</code>, and instead periodically log vitals and health to a log file. As long as we do that frequently, it should be as good.</p> <p><em>Update: With Go 1.8 that was just released,</em> <code>net/http</code> <em>is 2.2MB.</em></p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">788K gopkg.in/alecthomas/kingpin.v2.a </span><span class="z-text z-plain">388K github.com/alecthomas/template.a </span></code></pre> <p>This is a big surprise, around 1MB for a nice-to-have POSIX flag parsing feature. We can drop that and use the standard library flags, or even read configuration from environment variables and do away with flags completely (which I can tell you also takes some space).</p> <p>Newrelic adds up another 1.3MB, so we can drop that as well:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">668K github.com/newrelic/go-agent.a </span><span class="z-text z-plain">624K github.com/newrelic/go-agent/internal.a </span></code></pre> <p>Zap can go too. We can use the standard way to log in Go:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">392K go.uber.org/zap/zapcore.a </span></code></pre> <p>Otto, being an embedded Javascript engine should be heavy, and we can confirm that:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">2.2M github.com/robertkrimen/otto.a </span><span class="z-text z-plain">312K github.com/robertkrimen/otto/parser.a </span><span class="z-text z-plain">172K github.com/robertkrimen/otto/ast.a </span></code></pre> <p>Meanwhile, <code>logrus</code> is lightweight for being a feature-packed logging library:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">128K github.com/Sirupsen/logrus.a </span></code></pre> <p>We can leave that in.</p> <h3 id="take-aways">Take Aways</h3> <p>We found a way to break down Go dependency sizes and we saved around 7MB by accepting that we don’t <em>have</em> to use certain dependencies and that we can take alternative from Go’s standard library in their stead.</p> <p>I can tell you already that instead of these 12MB we had, you can manage to squeeze this binary down to 1.2MB with some additional mix-and-match dependency action.</p> <p>For the general rule — you shouldn’t be doing this because Go dependencies are already small in comparison to other platforms. However, you should always make sure you have the tools that help you generate more visibility into the what you’re building, and <code>gofat</code> can be one of those tools when you're building for resource constrained environments.</p> <p><em>PS: if you want to test things out, here’s</em> <a href="https://github.com/jondot/fattyproject"><em>the reference Github repo</em></a> <em>.</em></p> <blockquote> <p><a href="http://bit.ly/Hackernoon">Hacker Noon</a> is how hackers start their afternoons. We’re a part of the <a href="http://bit.ly/atAMIatAMI">@AMI</a> family. We are now <a href="http://bit.ly/hackernoonsubmission">accepting submissions</a> and happy to <a href="mailto:partners@amipublications.com">discuss advertising &amp;sponsorship</a> opportunities.</p> </blockquote> <blockquote> <p>To learn more, <a href="https://goo.gl/4ofytp">read our about page</a>, <a href="http://bit.ly/HackernoonFB">like/message us on Facebook</a>, or simply, <a href="https://goo.gl/k7XYbx">tweet/DM @HackerNoon.</a></p> </blockquote> <blockquote> <p>If you enjoyed this story, we recommend reading our <a href="http://bit.ly/hackernoonlatestt">latest tech stories</a> and <a href="https://hackernoon.com/trending">trending tech stories</a>. Until next time, don’t take the realities of the world for granted!</p> </blockquote> How to Snapshot Test Everything in Your Redux App With Jest Wed, 07 Dec 2016 00:00:00 +0000 Unknown https://rng0.io/blog/how-to-snapshot-test-everything-in-your-redux-app-with-jest/ https://rng0.io/blog/how-to-snapshot-test-everything-in-your-redux-app-with-jest/ <p><img src="https://rng0.io/blog/how-to-snapshot-test-everything-in-your-redux-app-with-jest/1.png" alt="" /></p> <p>Javascript <a href="https://medium.com/@ericclemmons/javascript-fatigue-48d4011b6fc4#.gi7et0qqb">fatigue</a> is something I always try to fight. Some times it feels extra painful because I’m measuring it against the intuition I’ve built based on my experience with different platforms such as the JVM, .NET, Ruby, and Go. Put aside the ever-growing list of Javascript single page application frameworks, competing modules, and competing view libraries, we still have a considerable fatigue from the process of choosing a testing stack.</p> <p>A testing stack covers running and rerunning tests reliably and quickly, integrating with coverage statistics, with tooling — by offering different reporters, mocking libraries, and test flavor (BDD is one), and in some occasions the level of integration tests — against real browsers, headless browsers, or a virtual DOM. Then, you can pick and choose from <a href="https://mochajs.org/">mocha</a>, <a href="https://jasmine.github.io/">Jasmine</a>, <a href="https://github.com/avajs/ava">ava</a>, <a href="https://github.com/gotwarlost/istanbul">Istanbul</a> and <a href="https://github.com/istanbuljs/nyc">nyc</a>, and <a href="https://github.com/substack/tape">tape</a> and more.</p> <p>One beautiful way to alleviate this fatigue that I can identify from the endless stream of intriguing projects in the Javascript world today, is what Facebook has done with Jest.</p> <h3 id="just-use-jest">Just Use Jest</h3> <p>If you remember <a href="https://facebook.github.io/jest">jest</a> from previous, older, encounters — you might have remembered a <a href="https://github.com/facebook/jest/issues/116">slow library</a> that’s not as widely adopted. During 2016, Facebook has transformed Jest to be a state-of-the art testing library, and had the guts to point us to fixture tests as a first-class-citizen in our testing approach (read: snapshot testing, more on that later).</p> <p>For those with bad Jest experience — forget about that experience. I feel that by now Jest is a different tool,but don’t take my word, pause to read about how Facebook tackled <a href="http://facebook.github.io/jest/blog/2016/03/11/javascript-unit-testing-performance.html">performance</a>, revamped developer experience <a href="https://facebook.github.io/jest/blog/2016/04/12/jest-11.html">here</a>, and <a href="https://facebook.github.io/jest/blog/2016/06/22/jest-13.html">here</a> and <a href="https://facebook.github.io/jest/blog/2016/10/03/jest-16.html">here</a>, and <a href="https://facebook.github.io/jest/blog/2016/07/27/jest-14.html">here</a>. And so, with Jest, you get the feeling developer experience is important.</p> <h2 id="that-fixture-thing">That Fixture Thing</h2> <p>From Jest 14, you could use <a href="https://facebook.github.io/jest/blog/2016/07/27/jest-14.html">snapshot testing</a>. Snapshot testing is a fitting name for plain old fixture testing, it works by dumping test state into a format Jest can compare (JSON), and performs test validation against previous test results automatically, by comparing files results. If a result don’t exist yet, it will get created. It’s a take on things like <a href="https://github.com/vcr/vcr">vcr</a>, but it works for everything. This article will show you how to use Jest for everything.</p> <p>Before that, you should be aware that there was <em>the</em> fixture discussion once, in the Ruby world. In the center was <a href="https://twitter.com/dhh">David Heinemeier Hansson</a> (aka <a href="https://medium.com/u/54bcbf647830">DHH</a>), creator of the Rails web framework and founder of <a href="https://basecamp.com/">Basecamp</a>. As the story goes, the Rails and Ruby community moved out of fixture based tests because of runtime slowness and maintenance overhead, so much that they called fixture testing an <a href="https://semaphoreci.com/blog/2014/01/14/rails-testing-antipatterns-fixtures-and-factories.html"><em>anti-pattern</em></a>. DHH <a href="https://twitter.com/dhh/status/461491174751342592">stood his ground</a>, and other people <a href="https://twitter.com/dhh/status/472072157821169664">responded the same</a>, it was a big deal and went on for a while, until years later — no one cared any more.</p> <p>On a personal note, I think DHH was right, and people on the other side of the discussion were right in <em>their own way</em>, but for me, like DHH, I make fixtures take a large part in my testing strategy — and it doesn’t matter what technology I use; if a fixture test harness isn’t there, I’ll build it. With Jest snapshots, it comes built-in.</p> <p>It’s possible that one day people will feel passionate and dispassionate about Jest snapshot testing, enough to resurface this discussion — but in the Javascript world rather than the Ruby world. At least now you have a field guide for this potential flamewar.</p> <h2 id="snapshot-testing">Snapshot Testing</h2> <p>The classic example for snapshot tests, from the <a href="http://facebook.github.io/jest/docs">Jest docs</a>, goes like this:</p> <p>There’s so much value in this: it will render the view using a new <em>test-renderer</em>, will dump it into an intermediate, diff-able format (JSON), and save a file (“snaps” it). Compared to left-hand-right-hand coding for assertions to validate a view, this is a complete win.</p> <p>What makes snapshot testing so legitimate for this case is the fact that React and Redux took a more functional oriented approach. A good number of primitives around us are pure, for example pure components, and reducers, and there are more. In these cases we have <em>input</em>, <em>process</em>, and <em>output</em>. It’s perfect for fixture testing, and manually asserting the output instead, is a waste of time.</p> <p>Let’s see how to test a Redux application, almost entirely with Jest.</p> <h3 id="actions">Actions</h3> <p>Here’s how to test an asynchronous action that perform an HTTP request through a promise (using <a href="https://github.com/mzabriskie/axios">axios</a>, mocked with <a href="https://github.com/mzabriskie/moxios">moxios</a>):</p> <p>We use <a href="https://github.com/arnaudbenard/redux-mock-store">redux-mock-store</a> which lets us assert on its internals, such as <em>getActions()</em>. The idea is to get to a JSON representation of the result of the action as fast as possible, and to bolt in place the store’s internal state in terms of the actions that it saw.</p> <p>Testing a <em>synchronous</em> action is the a simplified variant of the async case, which will be along the lines of:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">expect</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">store</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">dispatch</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">someAction</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">toMatchSnapshot</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <h2 id="views">Views</h2> <p>Views are the poster-child of Jest snapshot tests. To snapshot test a view, you’ll need to use the <em>react-test-renderer</em> module:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn add<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>dev</span> react-test-renderer</span> </span></code></pre> <p>Here is a self-contained example that shows how to test a <em>Foobar</em> view:</p> <h2 id="interaction">Interaction</h2> <p>To test interactivity, I would say ditch snapshot testing, and go with <a href="https://github.com/airbnb/enzyme">enzyme</a>. Don’t worry, you can <em>still</em> use Jest mocks to follow a “blessed” Jest stack. First, install enzyme and dependencies:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> yarn add<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>dev</span> enzyme react-dom</span> </span></code></pre> <p>Note that for the moment you can’t use Jest snapshots and enzyme tests in the same physical file due to renderers being in conflict. You might want to separate the two kinds of tests in any case, because you’ll maintain the two kinds in a different manner — interaction tests often requires an understanding of the flow and the surrounding state, while snapshot tests are simple input-process-output, almost declarative tests.</p> <p>Here’s how to test a component called <em>Fuzzy</em> that interacts on press in React Native:</p> <p>With enzyme we find the first touchable and press it. But you could use what ever <a href="http://airbnb.io/enzyme/docs/api/ShallowWrapper/find.html">selector strategy</a> you prefer.</p> <h2 id="reducers">Reducers</h2> <p>We’re back to functional land. This means testing will be easy; reducers are almost fun to test because they take a couple inputs, performs some processing, and return a predictable output. Sounds like another perfect case for snapshot tests (I use <a href="https://github.com/facebook/immutable-js">Immutable.js</a> here for the state):</p> <p>You might feel that this looks almost code-generated. And you‘d be right. There’s plenty of low-hanging fruit to grab here — how about just declaring an input, an action, and a subject? Let’s take a detour.</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">reducerTest</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">someState</span><span class="z-punctuation z-separator z-comma z-ts">,</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span></span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span></span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">someAction</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">subjectReducer</span><span class="z-meta z-brace z-round z-ts">)</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">reducerTest</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">anotherState</span><span class="z-punctuation z-separator z-comma z-ts">,</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span></span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span></span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">someAction</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">subjectReducer</span><span class="z-meta z-brace z-round z-ts">)</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">reducerTest</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">anotherState</span><span class="z-punctuation z-separator z-comma z-ts">,</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span></span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span></span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">anotherAction</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">subjectReducer</span><span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <p>Which you can then compact into:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">subjectReducer</span><span class="z-meta z-brace z-square z-ts">]</span></span>:<span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">someState</span><span class="z-punctuation z-separator z-comma z-ts">,</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span></span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span></span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">someAction</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">anotherState</span><span class="z-punctuation z-separator z-comma z-ts">,</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span></span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span></span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">someAction</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">anotherState</span><span class="z-punctuation z-separator z-comma z-ts">,</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span></span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span></span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">anotherAction</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">anotherReducer</span><span class="z-meta z-brace z-square z-ts">]</span></span>:<span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-keyword z-operator z-spread z-ts">...</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p>And if we can get rid of the function there, by specifying their parameters out-of-band:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">subjectReducer</span><span class="z-meta z-brace z-square z-ts">]</span></span>:<span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">someState</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">someAction</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">anotherState</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">someAction</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">param1</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">param2</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">anotherState</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">anotherAction</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">anotherReducer</span><span class="z-meta z-brace z-square z-ts">]</span></span>:<span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-keyword z-operator z-spread z-ts">...</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p>Given we have a registry of symbols, we can now turn this into a data file, where each string will be a lookup key into a live object (action, state, or reducer).</p> <pre data-lang="json" class="language-json z-code"><code class="language-json" data-lang="json"><span class="z-source z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>subjectReducer<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span></span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-punctuation z-section z-sequence z-begin z-json">[</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-meta z-sequence z-json"><span class="z-punctuation z-section z-sequence z-begin z-json">[</span><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>someState<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>someAction<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-section z-sequence z-end z-json">]</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-meta z-sequence z-json"><span class="z-punctuation z-section z-sequence z-begin z-json">[</span><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>anotherState<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>someAction<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> <span class="z-constant z-numeric z-integer z-decimal z-json">8</span><span class="z-punctuation z-separator z-sequence z-json">,</span> <span class="z-constant z-numeric z-integer z-decimal z-json">15</span><span class="z-punctuation z-section z-sequence z-end z-json">]</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-meta z-sequence z-json"><span class="z-punctuation z-section z-sequence z-begin z-json">[</span><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>anotherState<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>anotherAction<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-section z-sequence z-end z-json">]</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-punctuation z-section z-sequence z-end z-json">]</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>anotherReducer<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-punctuation z-section z-sequence z-begin z-json">[</span><span class="z-invalid z-illegal z-expected-sequence-separator z-json">.</span><span class="z-invalid z-illegal z-expected-sequence-separator z-json">.</span><span class="z-invalid z-illegal z-expected-sequence-separator z-json">.</span><span class="z-punctuation z-section z-sequence z-end z-json">]</span></span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></code></pre> <p>And of course, you can provide the values for the action (not using an action creator function) and state (not using a pre-made state variable).</p> <p>Now you can have a proper test harness, without a single line of manual test set up.</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">reducerTests</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>reducer-tests.json<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">runTests</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">reducerTests</span><span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <p>This is a breeze through what’s possible. Here’s an open challenge — make a library out of this!</p> <h3 id="a-reducer-test-gotcha">A Reducer Test Gotcha</h3> <p>When a reducer is a big ball of a black-box switch/case, you don’t know what actions trigger it and what actions don’t. It may be that when you cover all of the actions you <em>know of</em> in tests, one action slips through because, well, people. And since you’re only testing what you know — this can trigger bad behavior in production.</p> <p>And so, we need to find a way to assert that a reducer does its job <em>and</em> responds to a closed set of actions, and these actions alone. You can do this with <a href="http://flowtype.org">Flow</a> using <a href="https://github.com/fbsamples/f8app/blob/master/js/actions/types.js">type annotations</a>; but in either case I’m not happy about big switch cases because I think their lack of modularity doesn’t fit with what Redux is trying to do.</p> <p>This is why I compose my reducer out of handlers using <a href="https://github.com/jondot/redux-duet">Duet</a>:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-keyword z-control z-export z-ts">export</span> <span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">reducerMap</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-class z-builtin z-ts">Object</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">assign</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">rehydrateHandler</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">statusActionHandler</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">requestActionHandler</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">saveUserHandler</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">deleteUserHandler</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-brace z-round z-ts">)</span></span> </span><span class="z-source z-ts"><span class="z-meta z-export z-default z-ts"><span class="z-keyword z-control z-export z-ts">export</span> <span class="z-keyword z-control z-default z-ts">default</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">createReducer</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">initialState</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">reducerMap</span><span class="z-meta z-brace z-round z-ts">)</span></span> </span></code></pre> <p>Being that the map is exported, I can easily snapshot test it:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">expect</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">reducerMap</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">toMatchSnapshot</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <p>And it closes over the set of actions that the reducer responds to, and only to that set.</p> <h2 id="connect-and-selectors">Connect and Selectors</h2> <p>We’ve tested the bits that make a Redux app. But we need to be careful not to neglect the bits that connect the bits — the Redux connect function, mapping functions, and selectors. I like to place and export the Redux connect logic and <a href="https://github.com/reactjs/reselect">selectors</a> in a separate file, and in the same time export the smart component it connects — without connecting it.</p> <p>By now I’m guessing you’ve got the idea: to snapshot test, find a way to translate these into JSON. For the case of selectors — that’s easy. Its practically what they do — provide state and props to a function and it spits out another form of state. For a selector called <em>getUsers</em> this is it:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">expect</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">getUsers</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">state</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">props</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">toMatchSnapshot</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <p>For the mapping functions in <em>connect</em>, testing may be tricky. You can assert that mapping is in place by generating a snapshot, which will capture the names of the functions because what ever mapping is doing, it’s generating a map. But to assert that a function called <em>doFooBar</em> dispatches doFooBarAction — you’ll need to invoke it. This is a good motivation to find a declarative way to take care of that bit — or leave it to end-to-end tests or integration snapshot tests — where you connect everything, interact, and snapshot everything.</p> <h2 id="libraries">Libraries</h2> <p>Finally, to test a library — if you can position it to spit out anything that you can convert to JSON, you can snapshot test it.</p> <h2 id="finding-the-right-balance">Finding the Right Balance</h2> <p>If you take this approach, you’ll have a code base that will be composed of:</p> <ol> <li>Testing with snapshots.</li> <li>Interaction tests.</li> <li>Sniping tests.</li> </ol> <p>Being that a sniping test is a test that you’d only like to snipe a few values, properties, or make a few assertions. These would be much less rigid than snapshot tests (snapshots will break on <em>any</em> change).</p> <p>You’ll have to balance (1) and (3) and shuffle around how you want to write your tests. Mostly I’d say — don’t be afraid to go with snapshot tests for everything, because replacing those with regular, sniping tests, will be easier than the other way around.<a href="https://goo.gl/4ofytp"><img src="https://rng0.io/blog/how-to-snapshot-test-everything-in-your-redux-app-with-jest/4.png" alt="" /></a></p> <blockquote> <p><a href="http://bit.ly/Hackernoon">Hacker Noon</a> is how hackers start their afternoons. We’re a part of the <a href="http://bit.ly/atAMIatAMI">@AMI</a> family. We are now <a href="http://bit.ly/hackernoonsubmission">accepting submissions</a> and happy to <a href="mailto:partners@amipublications.com">discuss advertising &amp; sponsorship</a> opportunities.</p> </blockquote> <blockquote> <p>If you enjoyed this story, we recommend reading our <img src="https://rng0.io/blog/how-to-snapshot-test-everything-in-your-redux-app-with-jest/5.jpeg" alt="latest tech stories" /></p> </blockquote> React Native: Why and How to Build Your Native Code in Go Thu, 01 Dec 2016 00:00:00 +0000 Unknown https://rng0.io/blog/react-native-why-and-how-to-build-your-native-code-in-go/ https://rng0.io/blog/react-native-why-and-how-to-build-your-native-code-in-go/ <p>React Native allows us to build mobile apps, using Javascript. To me it’s the first platform that actually delivers on that promise; it does this through a mindful architecture, best-in-class tooling and workflow, and a pragmatic approach for cross-platform development. It didn’t state its purpose is to provide an ability to write 100% Javascript code for any given project, nor did promise a perfect, single, cross-platform codebase for your mobile apps. To me, that is the entire difference.</p> <p>And so, the idea of perfect cross-platform code sharing and an app that’s built entirely in Javascript is inherently tricky, because:</p> <ol> <li>Some parts you have to build natively.</li> <li>Different platforms are different in experience.</li> <li>Different devices perform differently.</li> <li>There’s <a href="https://en.wikipedia.org/wiki/No_Silver_Bullet">no silver bullet</a>.</li> </ol> <p>This means that ultimately we’re all bound to write <em>some</em> native code. Writing native code is a radical shift from our React Native experience: we have to know the tooling more intimately; Xcode, Android Studio and Gradle and the Android SDK, Obj-C or Swift, and Java or any Android-friendly JVM language such as Kotlin. We also need to get familiar with each platform’s framework mental model, such as Android’s Intents and Views, and Cocoa’s controllers, views and delegates.</p> <h3 id="why-go-native">Why Go Native?</h3> <p>But before that, even when on React Native, we can list real reasons that force us to ditch Javascript, and go native.</p> <ul> <li><strong>Performance</strong>. This is arguably the first item that crosses everyone’s mind when they say “native code”, and it’s also the first item you should work hard to rule out. You have to first prove, through numbers, that you have a performance problem. And then I’d remain suspicious if I were you. Still, native code trumps Javascript performance by a good degree for certain workloads.</li> <li><strong>Low level OS access</strong>. This is already all around us. We use React Native packages that deal with the <a href="https://github.com/johanneslumpe/react-native-fs">native parts</a> of each OS, because core React Native hasn’t got to it yet, or it’s out of scope.</li> <li><strong>Infrastructure and robustness</strong>. There are tasks that are wasteful to do on the Javascript engine such as processing images, moving large amounts of bytes across the React Native bridge, <a href="https://github.com/facebook/react-native/issues/854">processing files</a> and more.</li> <li><strong>Tricky async logic</strong>. We might prefer using the <a href="https://www.raywenderlich.com/60749/grand-central-dispatch-in-depth-part-1">native OS facilities</a> for dealing with async workflows, or prefer the “local” language constructs for doing so. Different mental models for concurrency like <a href="http://golang.org">CSP</a> and the <a href="http://akka.io/">actor model</a> come to mind.</li> <li><strong>Libraries and ecosystem</strong>. Some libraries might not be available on npm, or don’t have the quality we expect, or features, or community support. The native variant might be already gold-standard.</li> <li><strong>Obfuscation and resistance to reverse engineering</strong>. React Native Javascript code ships with your binary, and anyone can grab it from your package. While some platforms such as Xamarin encrypt and embed it in native code, <a href="https://www.blackhat.com/docs/asia-15/materials/asia-15-Grassi-The-Nightmare-Behind-The-Cross-Platform-Mobile-Apps-Dream.pdf">it’s still there (PDF)</a> for an attacker to pry open. In most cases it isn’t a concern but there are plenty of cases where your IP is actually your code.</li> <li><strong>Code sharing with a backend</strong>. While one of the major benefits of using <a href="https://nodejs.org">Node.js</a> is sharing the backend and frontend Javascript code, it may be that your backend isn’t built on Node.js nor Javascript, and you’d still want to share artifacts such as data model, validation and business rules.</li> <li><strong>Lastly, my favorite: binary communication</strong>. While you could communicate over a binary protocol such as <a href="https://github.com/google/protobuf">Protobuf</a> from your React Native process, it may be painful and challenging to unpack binary data, or for that to perform well, and for that to be maintainable or supported by existing platform and libraries.</li> </ul> <p>As React Native evolves, some of the items in this list will disappear, and still some, like obfuscation and others, will always remain.</p> <h3 id="using-go-for-cross-platform-mobile-development">Using Go for Cross-Platform Mobile Development</h3> <p>Ruling out guest VMs that, as Javascript, host a language on a given platform, or commercial engines like <a href="http://www.apportable.com/">Apportable</a>, we’re left with C and C++ (dropbox used <a href="https://oleb.net/blog/2014/05/how-dropbox-uses-cplusplus-cross-platform-development/">C++ for sharing non-UI code</a>) as target languages for native code sharing.</p> <p>But these are not safe (as in memory safety), and in comparison to Javascript, are hard core. Using Java, Obj-C or C/C++ may create a considerable amount of friction. There’s also Rust, but I’m not sure how well it plays with ARM, and what kind of mobile abstractions it has (none?).</p> <p>Up until a couple years ago, that could have been where our experiment ended. Then, Go (the language) has added support to export binaries as C-libraries, which opened up a world of opportunities. For example, years ago I used Go to fix performance bottlenecks in Ruby, and later <a href="https://rng0.io/blog/ruby-and-go-sitting-in-a-tree/">wrote an article and a reference Gem implementation</a> to show how you could do it too.</p> <p>Another of which was putting Go on mobile devices. I’m going to show you that Go is as good a language to use for cloud infrastructure, as well as mobile development, and — to great extent — mobile development for React Native.</p> <h3 id="our-prisma-clone-primer">Our Prisma Clone: Primer</h3> <p>We’ll make an app that render images like this:</p> <p><img src="https://rng0.io/blog/react-native-why-and-how-to-build-your-native-code-in-go/1.jpg" alt="We use Primitive by Michael Fogleman" /></p> <p>If you’re wondering about the image processing algorithm behind this image, it’s called <a href="https://github.com/fogleman/primitive">primitive</a>, by Michael Fogleman, and you can also buy a Mac app that he built <a href="https://primitive.lol/">here</a>.</p> <p>We’ll take this algorithm and build a mobile app around it, where the algorithm is originally written in Go.</p> <p><img src="https://rng0.io/blog/react-native-why-and-how-to-build-your-native-code-in-go/2.png" alt="" /></p> <p>And this is how it will looks like live (video is on fast-speed):</p> <p><img src="https://rng0.io/blog/react-native-why-and-how-to-build-your-native-code-in-go/3.gif" alt="" /></p> <p>Since the core algorithm is native, and built in Go, we’ll use it as-is. But to be clear, if we had to build this algorithm or any other algorithm or piece of infrastructure that must be native, and we didn’t have an existing codebase already, we had a hard decision to make:</p> <ul> <li>Build it twice in Java and Obj-C.</li> <li>Once in C/C++.</li> <li>Once in Go.</li> </ul> <p>Actually, for this particular use case and algorithm, we tick even more items from the “why go native” list above:</p> <ol> <li><strong>Performance</strong>. Go will exhibit better performance than Javascript (we’ll see by how much later). We need it here, since <a href="https://github.com/fogleman/primitive">primitive</a> is a demanding cpu-based algorithm. Actually, it’s so demanding that it makes processing these images on a strong iPhone slow. Go is closer to the metal here and makes a good case to use it instead of Javascript.</li> <li><strong>Using a library that doesn’t exist on npm</strong>. And this time, because it’s a novel algorithm implemented in Go, and there’s no port of that any where else, I was happy that Go on mobile (and later, I made it work on React Native) works so well.</li> <li><strong>Obfuscation</strong>. In this case the target Go library we’re using is open source. If it weren’t the case, and you cared about your IP, no one could reverse engineer this algorithm without investing an unreasonable amount of time.</li> <li><strong>Native API access</strong>. This algorithm can save progress snapshots to disk on every iteration. If this were to be Javascript, we’d have to encode each snapshot and send it over as a big base64 string across the React Native bridge.</li> </ol> <p>And of course, we can use the same <em>native code</em> for both Android and iOS.</p> <h3 id="the-go-mobile-toolchain">The Go Mobile Toolchain</h3> <p>If you’re already familiar with Go as a language, then this will be a nice fresh breeze. If not, to get familiar with Go, check out <a href="http://tour.golang.org/">the Go tour</a>.</p> <p>After you’ve done that, let’s install and exercise the gomobile toolchain (I assume you’ve installed Go and were able to do a simple “hello world” with it in the meanwhile):</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> npm install<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>g</span> ios-deploy</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> go get golang.org/x/mobile/cmd/gomobile</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> gomobile init <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> it might take a couple minutes</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> gomobile build<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>target</span><span class="z-keyword z-operator z-assignment z-option z-shell">=</span>ios golang.org/x/mobile/example/basic</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> ios-deploy<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>b</span> basic.app</span> </span></code></pre> <p>If everything worked well, you should be seeing a mobile app in your simulator. What this mobile toolchain did is build a native iOS library, generate a skeleton app, link these, and package an app. It all happened quietly (or humbly?), that’s part of the Go philosophy: success should be silent; in other words — no news is good news.</p> <p>I encourage you to read the short <a href="https://github.com/golang/go/wiki/Mobile">Wiki article</a>. To get proper context about these tools.</p> <h3 id="connecting-react-native-and-go-mobile">Connecting React Native and Go Mobile</h3> <p>We start by making a fresh React Native app, and grab the react-native-camera <a href="https://github.com/lwansbrough/react-native-camera/blob/master/Example/Example.js">example here</a>. This is a convenient way to get to the point of this article but you may want to build your own from scratch.</p> <p>We want to build our native Go library now and start by connecting it to our iOS app. This is where it might become tricky because we need to think about our workflow before everything else.</p> <p>In Go, the best thing to do is develop your Go code in your GOPATH, which is an opinionated place where your Go code <em>and</em> third-party libraries live (there’s a solid logic around that, but it’s out of the scope of this article). And so our first decision is to put our native library there as well — and not alongside our app.</p> <p>Once nice bonus is that our new Go package is ignorant of the fact that we’ll use it on a mobile device. In fact, you can conveniently work on it as a stand-alone Go app and then decide to bind it to your mobile app when you’re ready.</p> <p>For me, it lives here (yours will be different):</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">&lt;home&gt;/workspaces/golang/src/github.com/jondot/primer-bind </span></code></pre> <p>The layout is simple:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">➜ primer-bind tree . </span><span class="z-text z-plain">. </span><span class="z-text z-plain">└── primer </span><span class="z-text z-plain"> └── prime.go </span><span class="z-text z-plain"> </span><span class="z-text z-plain">1 directory, 1 file </span></code></pre> <p>Let’s take a look at our Go code:</p> <p>A quick pass shows that we export OnIterationDone which is a callback like interface we’ll use from each native side and Process which is the entry point to our algorithm. There’s nothing special to do about types passing in and out; go takes care of generating code for marshalling these to the respective mobile platforms.</p> <p>For Process, I took the CLI code from primitive and hacked it so it would be a function I can export. We also export Bench and JsonDummy for our benchmarking showcase.</p> <p>You’ll see that in each step in this article, I take the smallest possible step to strategically create code that we can discuss without too much context.</p> <p>Once this is in place, we have to switch context and cd to our app. We now want to make use of the gomobile toolchain, so we’ll run this:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">gomobile</span></span><span class="z-meta z-function-call z-arguments z-shell"> bind<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>x</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>v</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>target</span><span class="z-keyword z-operator z-assignment z-option z-shell">=</span>ios github.com/jondot/primer-bind/primer</span> </span></code></pre> <p>It will generate a native library for you and spit a lot of detail while doing so because we’re using -x -v; but you can drop these if you like. You can see the beauty of treating every Go code like a package: we supply a path that is both a repo and a local path in your GOPATH, if you didn’t have a local copy, Go would fetch it and build using it, or leave this as-is and it’ll grab my package and compile it so that you wouldn’t have to.</p> <p>Next up, we’ll have to build a bridge in our native code to glue everything together.</p> <p>Here, we’re making an RCTEventEmitter because we want to get notified where the native Go algorithm finishes its work. We also wrap our OnIterationDone struct with a nicer Obj-C interface. For where headers sit, it correlates with our Go package (we chose primer).</p> <p>Observation: this is how you would build a native module regardless of Go. We want to have an async function on the native side that gets variables such as iteration count, size, mode and number of workers and a callback, pass these to our Process function, here called GoPrimerProcess which is automatically generated for us, again it correlates with package name and function name, prefixed by “Go”.</p> <p>That’s basically it. The Javascript part now kicks in, and again we’ll do this like any other native module glue:</p> <h3 id="android">Android</h3> <p>For Android, this will be even easier because we have Android Studio support through a Gradle plugin. Remember, we still have our same Go package implemenation, untouched, in the same place. We need to specify that location in Gradle and wire the Android native bridge in Java.</p> <p>These would be the same steps to do in any Android and Go app. For more information you can look at the <a href="https://github.com/golang/mobile/tree/master/example/bind/android">reference project</a> and <a href="https://github.com/golang/mobile/tree/master/misc/androidstudio">here</a>.</p> <p>Like every Android native component in React Native, we have our module which contains the interesting bits and the package which instructs React Native to inject our module in.</p> <p>Now, our bridge code. The module part:</p> <p>And the package part:</p> <p>Our Java module code shows that the Go package magically appears in go.primer and under a pseudo-object called Primer (how convenient!). Note that gomobile generates the Go side for you and is idiomatically lays it like a real Java module.</p> <p>This completes a grand-tour around: Go, iOS (bridge), Javascript (glue), and Android (bridge).</p> <h3 id="a-grand-tour">A Grand Tour</h3> <p>This has been a tour around four languages, doing plumbing work: In Go, we wired the real Go module to make a function which we can export cleanly. On iOS, we built a native module and bridged it to the native library, which isn’t different than any other C library for that matter. In Javascript, we wired the native part to application-side code which isn’t unlike anything we’d do for a regular native module. Closing up, on Android, we set up a nice build configuration and a simple native module bridge.</p> <p>Most of this investment in plumbing work would be the same if we were to build any native module. The difference here is that now we have a single native, Go based, codebase to work with — and for the long-run this is going to save a ton of time because we don’t need to maintain the same parallel codebases on Android <em>and</em> iOS.</p> <h3 id="the-performance-question">The Performance Question</h3> <p>How does our code perform in Javascript compared to Go? We’ll run a silly benchmark, which probably is misleading for any real-life decision making so don’t take it as a fact and you can try it live in PrimerApp. TL;DR is that Go is 2x faster than Javascript for <a href="https://github.com/jondot/primer-bind/blob/master/primer/prime.go#L22">this</a> and <a href="https://github.com/jondot/PrimerApp/blob/master/Example.js#L109">this</a> benchmarks.</p> <h2 id="conclusion">Conclusion</h2> <p>Go with React Native is a <em>match made in heaven</em> for the following scenarios:</p> <ul> <li>You are passionate about Go or already have existing code in Go, or want to share backend Go code with your mobile app. Granted, specifically for that, you also have <a href="https://github.com/gopherjs/gopherjs">GopherJS</a> which works exceptionally well.</li> <li>You need the performance margin.</li> <li>You want the infrastructural robustness and type safety values of Go.</li> <li>You want proprietary code to stay safe.</li> <li>You don’t want to express ideas in <em>both</em> Obj-C/Swift and Java.</li> </ul> <p>There are certainly more motivation to match Go with React Native but I believe this small list was what you’ll meet day to day. With that, I hope you enjoyed this fusion of technologies and that you’ll come to build something interesting with the ideas I showed you here.</p> <p>You can find the reference app <a href="https://github.com/jondot/PrimerApp">here</a> and the Go module <a href="https://github.com/jondot/primer-bind">here</a>.</p> <p>Also, I’m passionate about obliterating native code duplication. If you need help doing this move, drop me a line at <a href="mailto:dotan@paracode.com">dotan@paracode.com</a>.<a href="https://goo.gl/4ofytp"><img src="https://rng0.io/blog/react-native-why-and-how-to-build-your-native-code-in-go/6.png" alt="" /></a></p> <blockquote> <p><a href="http://bit.ly/Hackernoon">Hacker Noon</a> is how hackers start their afternoons. We’re a part of the <a href="http://bit.ly/atAMIatAMI">@AMI</a> family. We are now <a href="http://bit.ly/hackernoonsubmission">accepting submissions</a> and happy to <a href="mailto:partners@amipublications.com">discuss advertising &amp; sponsorship</a> opportunities.</p> </blockquote> <blockquote> <p>If you enjoyed this story, we recommend reading our <img src="https://rng0.io/blog/react-native-why-and-how-to-build-your-native-code-in-go/7.jpeg" alt="latest tech stories" /></p> </blockquote> Redux Stack: Modular Redux Configuration Mon, 14 Nov 2016 00:00:00 +0000 Unknown https://rng0.io/blog/redux-stack-modular-redux-configuration/ https://rng0.io/blog/redux-stack-modular-redux-configuration/ <p>As the Redux ecosystem grows, libraries integrate by offering their own middleware, or enhancers, or even custom <strong>compose</strong> and <strong>createStore</strong> functions.</p> <p>When each library require a different way, or simply its own way to integrate with your Redux store, reducers, and middleware, it’s inevitable that your store set up code will grow and be more complex. This increases the risk that something will break, or at least for the codebase to be rigid and not modular enough to scale in the future.</p> <p><img src="https://rng0.io/blog/redux-stack-modular-redux-configuration/1.png" alt="" /></p> <h2 id="redux-stack">Redux Stack</h2> <p><a href="https://github.com/jondot/redux-stack"><strong>redux-stack</strong></a> is a library that helps you build modular, structured, and cleaner redux apps by making the “plugin” idea first-class.</p> <p>We introduce a concept of <em>initializers</em>: small pieces of integration code, per library, that “declares” how it integrates. <strong>redux-stack</strong> meshes these together to create your own store builder.</p> <h3 id="redux-initializers">Redux Initializers</h3> <p>An <em>initializer</em> in <strong>redux-stack</strong>, is a simple Javascript module that exports the following shape:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-export z-default z-ts"><span class="z-keyword z-control z-export z-ts">export</span> <span class="z-keyword z-control z-default z-ts">default</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-export z-default z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">reducers</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-export z-default z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> <span class="z-variable z-other z-readwrite z-ts">enhancers</span>:<span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-export z-default z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> <span class="z-variable z-other z-readwrite z-ts">composers</span>:<span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-export z-default z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"></span><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></code></pre> <p>The <strong>reducers</strong>, <strong>enhancers</strong> (remember, <strong>applyMiddleware</strong> is one example of an enhancer), and <strong>composers</strong> fields all contain one or more of the standard redux <a href="http://redux.js.org/docs/basics/Reducers.html">reducer</a>, <a href="https://github.com/reactjs/redux/blob/master/docs/Glossary.md#store-enhancer">enhancer</a> and <a href="https://github.com/reactjs/redux/blob/master/docs/api/compose.md">composer</a>.</p> <p>Granted, store enhancers are the blessed way to go forward for doing store customization, in reality, I bumped into libraries wanting to add their own reducer, or supply their own createStore function, or even require that you use their own version of a compose function. It was common enough to make these part of the initializer shape.</p> <p>Let’s create an initializer for <a href="https://github.com/exponentjs/ex-navigation">ex-navigation</a>: a popular navigation library for React Native that integrates well with Redux. Here’s the required configuration, verbatim from the README:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-variable z-other z-readwrite z-alias z-ts">createNavigationEnabledStore</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-alias z-ts">NavigationReducer</span> <span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>@exponent/ex-navigation<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-variable z-other z-readwrite z-alias z-ts">combineReducers</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-alias z-ts">createStore</span> <span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">createStoreWithNavigation</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">createNavigationEnabledStore</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">createStore</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">navigationStateKey</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>navigation<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">store</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">createStoreWithNavigation</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-comment z-block z-ts"><span class="z-punctuation z-definition z-comment z-ts">/*</span> combineReducers and your normal create store things here! <span class="z-punctuation z-definition z-comment z-ts">*/</span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">combineReducers</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">navigation</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-readwrite z-ts">NavigationReducer</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> other reducers</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-export z-default z-ts"><span class="z-keyword z-control z-export z-ts">export</span> <span class="z-keyword z-control z-default z-ts">default</span> <span class="z-variable z-other z-readwrite z-ts">store</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <p>A few observations: it uses its own <strong>createStore</strong> function, called <strong>createStoreWithNavigation</strong>. Next, it also wants to inject a reducer. This is completely trivial to do, however it may become tricky when another library wants to supply its own version of <strong>createStore</strong>, for example. And in the bigger sense of things — this is really special enough to tuck away and not bother us every time we’re looking at store configuration.</p> <p>Let’s turn that into an initializer compatible with redux-stack.</p> <p>First, a sharp eye would identify that <strong>createStoreWithNavigation</strong> can be turned into a standard Redux enhancer like so:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts"><span class="z-entity z-name z-function z-ts">navEnhancer</span></span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">createStoreFn</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">createNavigationEnabledStore</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">createStore</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-readwrite z-ts">createStoreFn</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">navigationStateKey</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>navigation<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span></span> </span></code></pre> <p>Now that we have a standard Redux enhancer, we can make the initializer:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-variable z-other z-readwrite z-alias z-ts">NavigationReducer</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-alias z-ts">createNavigationEnabledStore</span> <span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>@exponent/ex-navigation<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts"><span class="z-entity z-name z-function z-ts">navEnhancer</span></span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">createStoreFn</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">createNavigationEnabledStore</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">createStore</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-readwrite z-ts">createStoreFn</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">navigationStateKey</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>navigation<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-export z-default z-ts"><span class="z-keyword z-control z-export z-ts">export</span> <span class="z-keyword z-control z-default z-ts">default</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-export z-default z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">name</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>ex-navigation<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-export z-default z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">reducers</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">navigation</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-readwrite z-ts">NavigationReducer</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-export z-default z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">enhancers</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">navEnhancer</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-export z-default z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></code></pre> <p>And let’s make a single <strong>index.js</strong> to consolidate all initializers:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">initNavigation</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>./ex-navigation<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-storage z-modifier z-ts">export</span> <span class="z-keyword z-control z-switch z-ts">default</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span> </span></span><span class="z-source z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-variable z-other z-readwrite z-ts">initNavigation</span> </span></span><span class="z-source z-ts"><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">]</span></span> </span></code></pre> <p>After this is done, we want to drop the initializer somewhere convenient, so we’ll add another folder to a standard Redux layout:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">actions/ </span><span class="z-text z-plain">components/ </span><span class="z-text z-plain">constants/ </span><span class="z-text z-plain">containers/ </span><span class="z-text z-plain">initializers/ &lt;--- </span><span class="z-text z-plain"> - ex-navigation.js </span><span class="z-text z-plain"> - index.js </span><span class="z-text z-plain">reducers/ </span><span class="z-text z-plain">store/ </span></code></pre> <p>Now, your store set up code can look nice and clean, like this:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-variable z-other z-readwrite z-alias z-ts">createStore</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-alias z-ts">combineReducers</span> <span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-variable z-other z-readwrite z-alias z-ts">buildStack</span> <span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux-stack<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">stack</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>@/initializers<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">initialState</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-object-binding-pattern-variable z-ts"><span class="z-punctuation z-definition z-binding-pattern z-object z-ts">{</span> <span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">reducers</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">enhancer</span></span> <span class="z-punctuation z-definition z-binding-pattern z-object z-ts">}</span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">buildStack</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">stack</span><span class="z-meta z-brace z-round z-ts">)</span></span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">store</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">createStore</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">combineReducers</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">reducers</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">initialState</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">enhancer</span><span class="z-meta z-brace z-round z-ts">)</span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-export z-default z-ts"><span class="z-keyword z-control z-export z-ts">export</span> <span class="z-keyword z-control z-default z-ts">default</span> <span class="z-variable z-other z-readwrite z-ts">store</span></span> </span></code></pre> <p>If you keep a clean structure, this block of code will be similar for every app that you build, and can be easily extracted into its own piece of library.</p> <h4 id="sharing-initializers">Sharing Initializers</h4> <p>We’ve seen how to refactor a Redux app into an app that uses <a href="https://github.com/jondot/redux-stack"><strong>redux-stack</strong></a>, which results in cleaner set up code and a more modular app over all. It means that instead of reading library set up instructions and copying code to your main store configuration helper every time you add a library, you create an initializer and drop it in the <strong>initializers/</strong> folder.</p> <p>Creating an initializer has a small price. However, you can now easily copy that initializer to another project, and it will work as expected there too. It is time well spent.</p> <p>To take the idea one step forward, you can even give this initializer to another team, or open-source it for others to enjoy. If you’re a library author, you can supply this initializer for your users as part of the library.</p> <h3 id="towards-redux-initializers-and-presets">Towards Redux Initializers and Presets</h3> <p>And so, one goal around <strong>redux-stack</strong> is to have an extensive, pre-made set of initializers as part of redux-stack, or libraries themselves, so people don’t need to make them. For every well-known library you’ll have a ready-made initializer that takes care of its configuration and integration into your Redux app in a uniform way.</p> <p>Another goal is to provide <em>presets</em>, which are pre-made sets of initializers. Or in other words: complete Redux app stacks for you to use. This idea is inspired from <a href="https://babeljs.io/docs/plugins/preset-latest/">babel presets</a>. So you could even do something like the listing below, to get <a href="http://exponentjs.com">Exponent</a>’s favorite flavor of infrastructure (note: I’m not affiliated with Exponent, I just appreciate their libraries):</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-variable z-other z-readwrite z-alias z-ts">createStore</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-alias z-ts">combineReducers</span> <span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-variable z-other z-readwrite z-alias z-ts">buildStack</span> <span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux-stack<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">stack</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>stack-exponent-react-native<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-object-binding-pattern-variable z-ts"><span class="z-punctuation z-definition z-binding-pattern z-object z-ts">{</span> <span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">reducers</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">enhancer</span></span> <span class="z-punctuation z-definition z-binding-pattern z-object z-ts">}</span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">buildStack</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">stack</span><span class="z-meta z-brace z-round z-ts">)</span></span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">store</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">createStore</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">combineReducers</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">reducers</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">enhancer</span><span class="z-meta z-brace z-round z-ts">)</span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-export z-default z-ts"><span class="z-keyword z-control z-export z-ts">export</span> <span class="z-keyword z-control z-default z-ts">default</span> <span class="z-variable z-other z-readwrite z-ts">store</span></span> </span></code></pre> <h3 id="see-the-big-picture">See the Big Picture</h3> <p>Here’s how Redux Stack improves the way your codebase feels, and supports modularity.</p> <h3 id="before">Before</h3> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">users</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>@/modules/users/state<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">settings</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>@/modules/settings/state<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">intro</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>@/modules/intro/state<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-variable z-other z-readwrite z-alias z-ts">NavigationReducer</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-alias z-ts">createNavigationEnabledStore</span> <span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>@exponent/ex-navigation<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">thunk</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux-thunk<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">createLogger</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux-logger<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-variable z-other z-readwrite z-alias z-ts">createStore</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-alias z-ts">applyMiddleware</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-alias z-ts">combineReducers</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-alias z-ts">compose</span> <span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-variable z-other z-readwrite z-alias z-ts">persistStore</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-alias z-ts">autoRehydrate</span> <span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux-persist<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">immutableTransform</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux-persist-transform-immutable<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-variable z-other z-readwrite z-alias z-ts">AsyncStorage</span><span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>react-native<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">devtools</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-support z-variable z-dom z-ts">window</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">__REDUX_DEVTOOLS_EXTENSION_COMPOSE__</span> <span class="z-keyword z-operator z-logical z-ts">||</span> <span class="z-variable z-other z-readwrite z-ts">compose</span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">reducers</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">users</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">settings</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">intro</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">rootReducer</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">combineReducers</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">navigation</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-readwrite z-ts">NavigationReducer</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-keyword z-operator z-spread z-ts">...</span><span class="z-variable z-other z-readwrite z-ts">reducers</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">logger</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">createLogger</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">collapsed</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-language z-boolean z-true z-ts">true</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> only log in development mode</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-entity z-name z-function z-ts">predicate</span></span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span><span class="z-variable z-other z-readwrite z-ts">__DEV__</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> transform immutable state to plain objects</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-entity z-name z-function z-ts">stateTransformer</span></span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">state</span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">removeNavigation</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">fromJS</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">state</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">toJS</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> transform immutable action payloads to plain objects</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-entity z-name z-function z-ts">actionTransformer</span></span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">action</span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-arrow z-ts"> </span><span class="z-variable z-other z-readwrite z-ts">action</span> <span class="z-keyword z-operator z-logical z-ts">&amp;&amp;</span> <span class="z-variable z-other z-object z-ts">action</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">payload</span> <span class="z-keyword z-operator z-logical z-ts">&amp;&amp;</span> <span class="z-variable z-other z-object z-ts">action</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">payload</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">toJS</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> <span class="z-keyword z-operator z-ternary z-ts">?</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-keyword z-operator z-spread z-ts">...</span><span class="z-variable z-other z-readwrite z-ts">action</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">payload</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">action</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">payload</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">toJS</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> <span class="z-keyword z-operator z-ternary z-ts">:</span> <span class="z-variable z-other z-readwrite z-ts">action</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">let</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">middleware</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-variable z-other z-readwrite z-ts">thunk</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-variable z-other z-readwrite z-ts">logger</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">]</span></span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">createStoreWithNavigation</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">createNavigationEnabledStore</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">createStore</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">navigationStateKey</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>navigation<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">initialState</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">store</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">createStoreWithNavigation</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">rootReducer</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">initialState</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">compose</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">autoRehydrate</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">devtools</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">applyMiddleware</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-keyword z-operator z-spread z-ts">...</span><span class="z-variable z-other z-readwrite z-ts">middleware</span><span class="z-meta z-brace z-round z-ts">)</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-brace z-round z-ts">)</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-brace z-round z-ts">)</span> </span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-brace z-round z-ts">)</span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">persistStore</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">store</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">storage</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-readwrite z-ts">AsyncStorage</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">blacklist</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>navigation<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">transforms</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">immutableTransform</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-export z-default z-ts"><span class="z-keyword z-control z-export z-ts">export</span> <span class="z-keyword z-control z-default z-ts">default</span> <span class="z-variable z-other z-readwrite z-ts">store</span></span> </span></code></pre> <p>This setup code is convoluted. In addition, imports and library usage are separated and are out of mind — they’re all over the place. It also means that if you ever want to change, or remove components you’ll have to hunt them down and hope you didn’t break your app; we can also see that there’s a lot of importance for how your order your integration.</p> <h3 id="after">After</h3> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-variable z-other z-readwrite z-alias z-ts">createStore</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-alias z-ts">combineReducers</span> <span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-variable z-other z-readwrite z-alias z-ts">buildStack</span> <span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>redux-stack<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">stack</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>@/initializers<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">initialState</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-object-binding-pattern-variable z-ts"><span class="z-punctuation z-definition z-binding-pattern z-object z-ts">{</span> <span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">reducers</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">enhancer</span></span> <span class="z-punctuation z-definition z-binding-pattern z-object z-ts">}</span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">buildStack</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">stack</span><span class="z-meta z-brace z-round z-ts">)</span></span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">store</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">createStore</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">combineReducers</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">reducers</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">initialState</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">enhancer</span><span class="z-meta z-brace z-round z-ts">)</span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-export z-default z-ts"><span class="z-keyword z-control z-export z-ts">export</span> <span class="z-keyword z-control z-default z-ts">default</span> <span class="z-variable z-other z-readwrite z-ts">store</span></span> </span></code></pre> <p>And we have a new initializers folder:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">initializers/ </span><span class="z-text z-plain">├── devtools.js </span><span class="z-text z-plain">├── ex-navigation.js </span><span class="z-text z-plain">├── index.js </span><span class="z-text z-plain">├── middleware.js </span><span class="z-text z-plain">├── reducers.js </span><span class="z-text z-plain">├── redux-logger.js </span><span class="z-text z-plain">└── redux-persist.js </span></code></pre> <p>The set up code is a no brainer. It is clean and reasonable and probably will look the same for every new app you make, so you can even extract that out into an npm module.</p> <p>The <a href="https://github.com/jondot/redux-stack/tree/master/examples/initializers">initializers realm is separate and modular</a>. You can remove and add any initializer you like and be certain nothing will break. If you built your app in a modular fashion, every module can “contribute” an initializer which sets itself up, and no drama happens — no other code has to change and no module to wreck by mistake.</p> <h3 id="getting-started">Getting Started</h3> <p>You can start using <strong>redux-stack</strong> today, the implementation behind it is no more than 30 lines of code, but the idea around uniform initializers being first-class is much more profound. Go ahead and <a href="https://github.com/jondot/redux-stack">check out the Github repo</a> for more information about using <strong>redux-stack</strong>.</p> <p><em>Thanks</em> <a href="https://medium.com/u/19a6a6d3d3f2"><em>Maiz Lulkin</em></a> <em>and</em> <a href="https://medium.com/u/dc0f4cb7debe"><em>Xavier Via</em></a> <em>for listening to my thoughts!</em></p> Debugging React Native Performance: Snoopy and the MessageQueue Thu, 10 Nov 2016 00:00:00 +0000 Unknown https://rng0.io/blog/debugging-react-native-performance-snoopy-and-the-messagequeue/ https://rng0.io/blog/debugging-react-native-performance-snoopy-and-the-messagequeue/ <p><em>This post is part of a series about React Native Performance, and the upcoming</em> <a href="http://programmingreactnative.com"><em>Programming React Native Vol 2</em></a> _._In my previous <a href="https://rng0.io/blog/debugging-react-native-applications/">Debugging React Native</a> article I mentioned the <strong>MessageQueue</strong>, the <strong>SPY_MODE</strong> flag, and how I submitted a pull request that changed it effectively into a <strong>spy()</strong> function anyone can try using. I also mentioned Snoopy, a tool I built that uses <strong>spy()</strong> and creates a high-level tool that helps you with debugging <strong>MessageQueue</strong> performance problems. Today, we’ll take a look at how that came to be, and how you can use Snoopy to figure out your performance problems.</p> <p>The React Native <strong>MessageQueue</strong> is the facility through which both native and Javascript parts communicate. As we adopt new libraries and components, they’ll be using the same facility to communicate, implicitly. Each library we add, in addition to each new piece of code we add, use the same pipe to send and receive messages to, and from, the native part.</p> <p><img src="https://rng0.io/blog/debugging-react-native-performance-snoopy-and-the-messagequeue/1.png" alt="" /></p> <p>Javascript, the bridge (and MessageQueue) and native parts</p> <p>Inevitably, some of the components we’ll adopt will make heavy use of the <strong>MessageQueue</strong>, but actually we’ll never know or have the time to notice. We perceive the <strong>MessageQueue</strong> as an internal thing and (for a good reason) and to be honest we tend to treat libraries as black boxes that frees our time and let us work on solving other, more interesting, problems.</p> <p>I bumped into my share of <strong>MessageQueue</strong> abuse. A year ago I’ve started building media apps with React Native, and I realized I had performance issues. Spending time profiling the Javascript side, I was certain there are no performance issues. Spending time on the native side, again, I was certain there were no performance issues. Frustrating.</p> <p>But here’s the deal. I had to realize the <strong>MessageQueue</strong> <em>existed</em>, I mean, I knew it existed — but I didn’t realize I should be looking there for answers. I found that bits of images were passing back and forth as base64 serialized values that were too heavy to go on the same channel that keeps the Javascript and native realms in sync.</p> <p>For that, I had to also realize how I would pry the <strong>MessageQueue</strong> open and see it overflowing with big messages and needless communication, due to the way I was using libraries at the time.</p> <h2 id="the-need-for-a-message-queue">The Need for a Message Queue</h2> <p>If you imagined you had to build React Native from scratch, how would that be? Let’s use the simplest module I could think of for illustration: <strong>Alert</strong>.</p> <p>First, ergonomically, it looks like a plain Javascript module:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">Alert</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">alert</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span><span class="z-source z-ts"> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>Title<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span><span class="z-source z-ts"> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>msg<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span> </span><span class="z-source z-ts"><span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <p>Behind the scenes, translating that to iOS would have to mean something along the lines of this:</p> <pre data-lang="swift" class="language-swift z-code"><code class="language-swift" data-lang="swift"><span class="z-source z-swift"><span class="z-storage z-type z-swift">let</span> alertController <span class="z-keyword z-operator z-assignment z-swift">=</span> UIAlertController(title: <span class="z-meta z-literal z-string z-swift"><span class="z-string z-quoted z-double z-swift">&quot;</span><span class="z-string z-quoted z-double z-swift">T</span><span class="z-string z-quoted z-double z-swift">i</span><span class="z-string z-quoted z-double z-swift">t</span><span class="z-string z-quoted z-double z-swift">l</span><span class="z-string z-quoted z-double z-swift">e</span><span class="z-string z-quoted z-double z-swift">&quot;</span></span>, message: <span class="z-meta z-literal z-string z-swift"><span class="z-string z-quoted z-double z-swift">&quot;</span><span class="z-string z-quoted z-double z-swift">m</span><span class="z-string z-quoted z-double z-swift">s</span><span class="z-string z-quoted z-double z-swift">g</span><span class="z-string z-quoted z-double z-swift">&quot;</span></span>, preferredStyle: <span class="z-keyword z-operator z-custom z-prefix z-unary z-swift">.</span>Alert) </span><span class="z-source z-swift"><span class="z-storage z-type z-swift">let</span> defaultAction <span class="z-keyword z-operator z-assignment z-swift">=</span> UIAlertAction(title: <span class="z-meta z-literal z-string z-swift"><span class="z-string z-quoted z-double z-swift">&quot;</span><span class="z-string z-quoted z-double z-swift">O</span><span class="z-string z-quoted z-double z-swift">K</span><span class="z-string z-quoted z-double z-swift">&quot;</span></span>, style: <span class="z-keyword z-operator z-custom z-prefix z-unary z-swift">.</span>Default, handler: <span class="z-constant z-nil z-swift">nil</span>) </span><span class="z-source z-swift">alertController<span class="z-keyword z-operator z-custom z-postfix z-unary z-swift">.</span>addAction(defaultAction) </span></code></pre> <p>This kind of translation, or wrapping, is one hint at the massive amount of work needed to create a framework like React Native. It means wrapping native code and exposing it as Javascript. It would also mean we need a Javascript engine embedded inside our native platform.</p> <p>And so, we stated that we have a native part and a Javascript part for every running React Native process. It sounds obvious, but when you factor in a scalable architecture, extensible design, sandboxed code, smart rendering pipelines, you’ll realize these would have to be two separate living organisms and that they need a form of efficient message passing to communicate.</p> <p>This kind of architecture exists everywhere around us, for example, we’ve had <a href="https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage"><strong>postMessage</strong></a> on the Web. I had to hand-wave a lot of detail to keep focus here, but be sure that in React Native, you have these two parts and the architecture is much detailed than what I described.</p> <p>When we work on performance issues, we’re looking for bottlenecks. For example, high CPU usage would probably mean one or two hot areas or functions, which you can find with profiling tools. Once fixed, you can be sure a couple more hot areas will pop up, but on average performance would improve.</p> <p>The <strong>MessageQueue</strong> presents an interesting variant of a bottleneck. One that introduces <em>congestion</em>. It’s implicit, like the internet connection we use, or the roads we use to commute. To take the internet connection analogy, if there’s a too high rate of traffic, you’ll experience slowness, and similarly, if we have a too high volume of traffic. So we have small messages with high rate, and big messages with low rate and some times the combination of the two; and the <strong>MessageQueue</strong> happens to connect the Javascript and native parts together.</p> <h4 id="prying-open-the-message-queue">Prying Open the Message Queue</h4> <p>Ultimately we want to have the following:</p> <ol> <li>Have better visibility, we move <strong>MessageQueue</strong> away from the <em>implicit</em> zone and into our comfort zone.</li> <li>Have a good way to analyze what we see, so we can make informed decisions.</li> </ol> <h2 id="better-visibility">Better Visibility</h2> <p>To tackle the first item, I had a plan: find a flag I can switch on, or suffer and build tracing into the <strong>MessageQueue</strong> myself. I bumped onto a flag React Native core contributors used for their own debugging purposes called <strong>SPY_MODE</strong>. But, you had to switch it on by editing the <strong>MessageQueue</strong> code.</p> <p>Actually, that’s how the <strong>spy()</strong> function came to be.</p> <p><img src="https://rng0.io/blog/debugging-react-native-performance-snoopy-and-the-messagequeue/2.png" alt="" /></p> <p>I wanted the <a href="https://github.com/facebook/react-native/pull/9160"><strong>spy()</strong> function</a> to add a convenient way to, well, spy on the MessageQueue from your own user-land code. Today, this means you can try it on any React Native app, and it doesn’t matter where you drop these couple lines of code:</p> <p>You’ll get this:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">2016-10-10 19:56:44.089 [info][tid:com.facebook.react.JavaScript] &#39;event!&#39;, { type: 1, </span><span class="z-text z-plain"> module: &#39;UIManager&#39;, </span><span class="z-text z-plain"> method: &#39;updateView&#39;, </span><span class="z-text z-plain"> args: [ [Getter/Setter], [Getter/Setter], [Getter/Setter] ] } </span><span class="z-text z-plain">2016-10-10 19:56:44.090 [info][tid:com.facebook.react.JavaScript] &#39;event!&#39;, { type: 1, </span><span class="z-text z-plain"> module: &#39;UIManager&#39;, </span><span class="z-text z-plain"> method: &#39;updateView&#39;, </span><span class="z-text z-plain"> args: [ [Getter/Setter], [Getter/Setter], [Getter/Setter] ] } </span></code></pre> <p>The reason that it’s not nice to look at so far, is because you’re getting the raw info event, which has the following shape:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">{ </span><span class="z-text z-plain"> type: int (0=N-&gt;JS | 1=JS-&gt;N) </span><span class="z-text z-plain"> method: string, </span><span class="z-text z-plain"> module: string, </span><span class="z-text z-plain"> args: object, </span><span class="z-text z-plain">} </span></code></pre> <p>A moment before we see how this shape is good for us, let’s try the default way to spy on events:</p> <p>Which give us:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">2016-10-10 20:03:53.954 [info][tid:com.facebook.react.JavaScript] JS-&gt;N : UIManager.updateView([85,&quot;RCTView&quot;,{&quot;opacity&quot;:1,&quot;transform&quot;:[1,0,0,0,0,1,0,0,0,0,1,0,15,0,0,1]}]) </span><span class="z-text z-plain">2016-10-10 20:03:53.954 [info][tid:com.facebook.react.JavaScript] JS-&gt;N : UIManager.updateView([93,&quot;RCTView&quot;,{&quot;opacity&quot;:0,&quot;transform&quot;:[1,0,0,0,0,1,0,0,0,0,1,0,20,0,0,1]}]) </span><span class="z-text z-plain">2016-10-10 20:03:53.968 [info][tid:com.facebook.react.JavaScript] N-&gt;JS : JSTimersExecution.callTimers([[73]]) </span><span class="z-text z-plain">2016-10-10 20:03:53.969 [info][tid:com.facebook.react.JavaScript] JS-&gt;N : UIManager.updateView([97,&quot;RCTView&quot;,{&quot;opacity&quot;:1}]) </span></code></pre> <p>With that, we’re done with the first issue.</p> <h2 id="better-analysis">Better Analysis</h2> <p>If you’re running this side by side with me, you’ll notice that your logs are streaming events constantly. Arguably, the best way to tame a stream of events is with a reactive framework like <a href="https://github.com/ReactiveX/rxjs"><strong>rxjs</strong></a> (or <strong>Kefir</strong>, or <strong>most</strong>, or <strong>flyd</strong>).</p> <p>So if we stream these into a pipe that aggregates and shapes the data, we can come to conclusions efficiently, come up with numbers that signify “good” or “bad”, and automate this whole process to create our own alerts. Sounds familiar? Well, not by chance, this is the same process you’d use to tame network traffic, and, for all intents and purposes the <em>communication</em> through the <strong>MessageQueue</strong> is another form of traffic.</p> <p>Let’s see how we can do the above using a tool I’ve built called <a href="https://github.com/jondot/rn-snoopy"><strong>Snoopy</strong></a> which was, in fact, the reason I added the <strong>spy()</strong> function in the first place :).</p> <p>Within your app, or a new app you create, or even the Snoopy <a href="https://github.com/jondot/rn-snoopy/tree/master/examples/Snoopful">demo app</a>, find a place to add the following:</p> <p>You now have events, and we’re hooked into the <strong>MessageQueue</strong>. It’s a fully primed stream, onto which you can tack <strong>Rx</strong> constructs as you see fit.</p> <p>Filter is one of these constructs. With the <strong>Snoopy</strong> filter helper, let’s see how we can start analyzing this stream:</p> <p>These should get you started when the stream is overwhelming. For example, filtering on <strong>createView</strong> is a great way to verify that a <strong>ListView</strong> does not create new cells when you were expecting it the least.</p> <p>Here’s how it should look like:</p> <p><img src="https://rng0.io/blog/debugging-react-native-performance-snoopy-and-the-messagequeue/3.gif" alt="" /></p> <p>To surface trends, we can use the built-in graphing and aggregation helpers that ships with <strong>Snoopy</strong> called <strong>bars</strong>, and <strong>buffer</strong>.</p> <p>Here’s how it should look live:</p> <p><img src="https://rng0.io/blog/debugging-react-native-performance-snoopy-and-the-messagequeue/4.gif" alt="" /></p> <h2 id="one-last-thing">One last thing</h2> <p>Did you notice the Yellowbox in the shot above?</p> <p>Although we defined two goals, it’s trivial to build an alerting system that would trigger when we mistakenly abuse the <strong>MessageQueue</strong>:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Trigger a yellobox (true) once the rate</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> is more than 100 messages per second.</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">bars</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">infoAry</span></span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span></span><span class="z-variable z-other z-object z-ts">infoAry</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">100</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-language z-boolean z-true z-ts">true</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span>)<span class="z-meta z-brace z-round z-ts">(</span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">events</span> </span><span class="z-source z-ts"><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">subscribe</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <h2 id="freestyle-it">Freestyle it</h2> <p>I picked rxjs, and not other, perhaps more pure and better performing reactive libraries out there although I did use those initially (namely: <strong>most</strong>). The main reason was that rxjs comes with rich <a href="http://reactivex.io/documentation/operators.html">operators</a> out of the box, and <a href="http://reactivex.io/rxjs/">extensive educational material</a>.</p> <p>Go ahead and try <a href="https://github.com/jondot/rn-snoopy">Snoopy</a> and think up new and interesting ways to shape and look the stream of messages that go through the <strong>MessageQueue</strong>. <a href="mailto:dotan@paracode.com">Let me know</a> what you think. It should be dead easy now.</p> Debugging React Native Applications Thu, 25 Aug 2016 00:00:00 +0000 Unknown https://rng0.io/blog/debugging-react-native-applications/ https://rng0.io/blog/debugging-react-native-applications/ <p>When doing application development with interactivity, UI/UX and state, on a resource-limited platform, these questions often come up:</p> <ul> <li>Debugging — how do you debug applications?</li> <li>Inspecting — how do you inspect the UI tree?</li> <li>Touchables — how to understand interactive surfaces?</li> <li>Platform — how to reason about the “magical things” that happen out of sight?</li> <li>Slowness — how to get visibility about what’s slow?</li> <li>Layout — Specifically, about React — how to reason about React performance and efficiency?</li> </ul> <p>This is an overview of what’s available to you within React Native and its ecosystem.</p> <h2 id="debugging">Debugging</h2> <p>Debugging is seamless. You can debug Javascript, step into native, back to Javascript without configuring anything, and relying on <a href="https://developer.chrome.com/devtools">Chrome devtools</a> pointed at a <a href="https://github.com/facebook/react-native/blob/master/local-cli/server/util/debugger.html">debug interface</a> that belongs to React Native.</p> <ol> <li>Cmd+Z (or shake) to bring up the dev menu.</li> <li>Select ‘Debug JS Remotely’.</li> <li>A fresh Chrome instance will pop up, which you can immediately use for setting breakpoints and stepping through your app’s Javascript code.</li> </ol> <p><img src="https://rng0.io/blog/debugging-react-native-applications/2.png" alt="" /></p> <p>Remember to turn off remote debugging when done, by accessing the developer menu again (Cmd+Z).</p> <p>Using <strong>console.log</strong>, or “console.log debugging” is also very common. It might seem a bit primitive, however with time <strong>console.log</strong> became a rather <a href="https://developer.mozilla.org/en-US/docs/Web/API/Console">powerful tool</a> for the Javascript developer, and it definitely isn’t equal to a dirty-feeling puts/print/cout/what-have-you.</p> <p>Even if React Native doesn’t support the more advanced <strong>time, group, table</strong> interface, it would pop a yellowbox on <strong>warn</strong>, for example. React Native promotes a very fast feedback cycle, and using <strong>console.log</strong> rhymes better with that, rather than stop-the-world debugging and having to context switch — getting out of the editor and using a completely different toolset.</p> <p>NOTE: There’s also <em>Systrace</em>, but given that it is quite low-level, we’ll cover in its own article.</p> <h3 id="reactotron">Reactotron</h3> <p>If you haven’t already, go ahead and try <a href="https://github.com/reactotron/reactotron">Reactotron</a> now. If you’re like me, you can’t get enough of command-line tools, and this one is one of the best you can use around React and Redux.</p> <p><img src="https://rng0.io/blog/debugging-react-native-applications/3.gif" alt="" /></p> <p>Reactotron plugs into your Redux pipeline as a middleware and lets you inspect, fiddle and trace it with a live ncurses based interface which is really fun to use.</p> <h2 id="inspecting">Inspecting</h2> <p>Much like in Chrome’s <a href="https://developers.google.com/web/tools/chrome-devtools">devtools</a> when inspecting the DOM for regular web pages, you can enable live inspection of React Native components.</p> <ol> <li>Cmd+Z (or shake) to bring up dev menu.</li> <li>Select Inspector.</li> <li>Tap on ‘Inspect’.</li> </ol> <p><img src="https://rng0.io/blog/debugging-react-native-applications/4.gif" alt="" /></p> <p>You can now proceed to select a component, drill down or up the UI tree hierarchy by tapping the breadcrumbs pieces, and verify positioning, margins, padding and more. You can use this to find overlapping areas, surfaces that block event bubbling, transparent surfaces that overflow the existing viewport, and more.</p> <h3 id="touchables">Touchables</h3> <p>Showing areas that are touchable (users can tap those) can be useful when the interface is seamless and does not expose a clear border region for each touchable.</p> <p>You can expose these with the inspector as well:</p> <ol> <li>Cmd+Z (or shake) to bring up dev menu.</li> <li>Select Inspector.</li> <li>Tap on ‘Touchables’.</li> </ol> <p><img src="https://rng0.io/blog/debugging-react-native-applications/5.gif" alt="" /></p> <p>You can then hide the inspector and continue interacting with your application while every touchable is exposed. Crushed border line, color depicts type of touchable:</p> <h3 id="spying-on-the-bridge">Spying on the Bridge</h3> <p>In a very high-level view, the architecture of React Native is divided into user code and platform code. User code, is typically your application Javascript code, and platform code is typically a mixture of Native and Javascript code as part of React Native. These two parts communicate over the React Native bridge.</p> <p><img src="https://rng0.io/blog/debugging-react-native-applications/6.png" alt="" /></p> <p>There’s a lot to gain by simply looking at these communications. You could find bottlenecks (by identifying too much chatter over the wire), resource hogs (by observing very large objects passing back and forth), bugs (by finding misplaced communication, forgotten timers, and so on) and more.</p> <p>React Native comes out of the box with a special <strong>SPY_MODE</strong> flag that you can turn on in its <strong>MessageQueue</strong> component; through which these kind of communications pass. However, this was not exposed to end-users, but for React Native developers. I wanted to change that with <a href="https://github.com/facebook/react-native/pull/9160">this pull request</a>, by allowing end users to plug in their own “spies”.</p> <p>I then proceeded to build a spying framework which you can use today called <a href="https://github.com/jondot/rn-snoopy">Snoopy</a>.</p> <p>And here’s how it looks like:</p> <p><img src="https://rng0.io/blog/debugging-react-native-applications/8.jpg" alt="Aggregating and Charting Events with Bar" /></p> <p>With Snoopy, you can build sophisticated <em>bridge event pipelines</em> using the powerful <a href="https://github.com/ReactiveX">Reactive Extensions</a> and <a href="https://en.wikipedia.org/wiki/Functional_reactive_programming">FRP</a> concepts that declaratively model <em>what</em> you want to observe and detect, instead of <em>how</em> you want to detect it.</p> <p>Here are some ideas for things you can observe over the bridge with Snoopy:</p> <ul> <li>Buffer events for one second, and raise an alert when the total events cross a threshold.</li> <li>Identify huge calls by looking at the arguments for each event.</li> <li>Keep an eye open by charting out the bridge communication volumes.</li> </ul> <p>Check out the <a href="https://github.com/jondot/rn-snoopy">Snoopy repository</a> for more.</p> <p>Update: I published an <a href="https://rng0.io/blog/debugging-react-native-performance-snoopy-and-the-messagequeue/">in-depth article</a> about using Snoopy!</p> <h3 id="slowlog">Slowlog</h3> <p>Many databases implement a feature called a “ <a href="http://dev.mysql.com/doc/refman/5.7/en/slow-query-log.html">Slow Query Log</a>” or simply “slowlog”, which is a file that the database uses to log very slow queries. An administrator can simply tail this file, and/or ship it to a log aggregation service that notifies her when a slow query happens.</p> <p>I wanted to have this same concept for React Native applications.</p> <p>With <a href="https://github.com/jondot/react-native-slowlog">react-native-slowlog</a>, you wire such a capability into your React components piecemeal, and get a high-level idea in real time of the slow things that happen around your app. It is not a fine-grained diagnostics tool, but rather a general watchdog that sits there and pops an alert when things are majorly slow.</p> <p>You can find out more about slowlog in the <a href="https://github.com/jondot/react-native-slowlog">react-native-slowlog repository</a>.</p> <h3 id="jseventloopwatchdog">JSEventLoopWatchdog</h3> <p>Here’s a neat idea: if you set a Javascript timer to fire every second, and on every tick you look at the wall clock to make sure that exactly one second has passed, then you can detect if your Javascript code is running late.</p> <p>This idea is exactly what the <strong>JSEventLoopWatchdog</strong> component in React Native core does. It Detects stalls in the React Native Javascript event loop by scheduling a no-op task every X amount of time, recording the wall clock time it executed at in reality, and compares it to the expected time. If we have a stall, there will be a difference in clock times. The bigger the stall the bigger the difference.</p> <p>To use <strong>JSEventLoopWatchdog</strong>, you supply a threshold time — the duration, in milliseconds, equal to the lag you’re willing to accept. When the lag is bigger, you’ll get console messages notifying you about it:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">JSEventLoopWatchdog</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-function z-ts">require</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>JSEventLoopWatchdog<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">JSEventLoopWatchdog</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">install</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">thresholdMS</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">200</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <p>The bad news is that this is still an <em>internal</em> tool React Native uses since it’s not currently usable if you’re out of the react-native project space. The good news is that I’ve extracted it into <a href="https://github.com/jondot/react-native-slowlog">react-native-slowlog</a> and adapted the internals so that it can run outside of the react-native core.</p> <p>To use <strong>JSEventLoopWatchdog</strong> on your own, you need to use the <em>watchdog</em> component of slowlog:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-variable z-other z-readwrite z-alias z-ts">watchdog</span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>react-native-slowlog/watchdog<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">watchdog</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">install</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">thresholdMS</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">50</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <p>Now, if anything is stalled for more than 50ms, you’ll get a notification in your logs.</p> <h3 id="react-perf-tools">React Perf Tools</h3> <p>Last but not least, remember you still have all the vanilla React tooling available. The official docs makes a better explanation than I would ever make here, so I’ll just point you to it:</p> <p><a href="https://facebook.github.io/react/docs/perf.html">React Perf Tools</a></p> <h3 id="summary">Summary</h3> <p>React Native comes with a large number of tools by any open-source cross-platform SDK standards (such as Cordova or Xamarin). You can use out of the box inspection and seamless debugging, and from my experience, the React Native core team is welcoming more tooling from the community.</p> <p>I also feel I’ve had a very welcoming hand introducing my <strong>MessageQueue</strong> patch into React Native core, along with help to conform to Facebook’s standards; with such a large, high traffic project, this is not to be taken for granted.</p> <p>I wish to thank <a href="https://medium.com/u/8207432cb3f5">Pieter De Baets</a>, <a href="https://medium.com/u/6eea920e2da6">Martin Konicek</a> and the React Native core team for doing such a great work with the community!</p> Announcing: React Native Katas Wed, 22 Jun 2016 00:00:00 +0000 Unknown https://rng0.io/blog/announcing-react-native-katas/ https://rng0.io/blog/announcing-react-native-katas/ <p>In the beginning, designers used Photoshop or Illustrator to design for digital media, be it Web or Mobile. They then went through a precision-critical task of <a href="https://daliakramer.files.wordpress.com/2011/05/slice-print-screen.jpg">slicing</a> their designs, in a way that required them to <em>imagine</em> what would be the correct slicing strategy, come development time.</p> <p>Developers took these slices, bitched and moaned, and needless to say, designers were slicing and dicing their designs again. Repeat until pixel perfect. People lived with this. The tensions were always there. For some, it was unacceptable.</p> <h2 id="the-sketch-era">The Sketch Era</h2> <p>Sketch came around and turned things upside down. Sketch connected with reality by showing that designers and developers could work in a more harmonious way. Suddenly, it wasn’t crazy for a developer to have Sketch installed locally anymore (it <em>was</em> an order of magnitude cheaper than Photoshop or Illustrator). In addition, Sketch opened up an <a href="https://github.com/sketchplugins/plugin-directory">ecosystem</a>, driven by its simplicity of design. Sketch also respected Mobile and made it a first-class citizen by allowing you to treat screen resolutions first-hand, amongst other perks.</p> <p>Still, even <em>that</em> wasn’t good enough. Designers suddenly needed to own CSS, Sass, and even <a href="https://zeplin.io/">code</a>. But, I still remember the browser wars, the era where CSS had a zillion different ways to break, depending on which browser and operating system combination you used. Today, that still holds true, but less so. Designers should not handle that; but they do. To do a proper design hand-off, a lot of designers are picking up front-end and CSS and other styling skills these days to varying degrees of success, but it is far from perfect.</p> <h2 id="design-hand-off-react-native-edition">Design hand-off: React Native Edition</h2> <p>React Native’s styling and React’s composition model, in addition to the developer experience, make the <em>perfect</em> toolchain a designer can use for design hand-off. There is only one CSS-like variant to learn. There are no sucky CSS parts. There is a <em>sane</em> layout mode (flex). It sort of feels like you’re doing HTML, <em>without the burdens of foundations and bootstraps.</em> The experience is very physical (does not feel like “coding”) and very simple, and the tooling is already baked.</p> <p>But how can you possibly have the guts to hand a cutting edge technology to non-technologists and have them not cut themselves, learn quickly, and be positive about the overall experience?</p> <p>I had to build something to answer that.</p> <h2 id="react-native-katas">React Native Katas</h2> <p>I feel that there is a whole world to be discovered about React Native styling and how designers approach it. I can only hope that is the first step toward discovering that world.</p> <p><a href="https://github.com/jondot/ReactNativeKatas">React Native Katas</a> (RNK from now on) is an interactive hands-on experience for designers and beginner developers alike. Built on (and runs in) React Native itself, as a designer — RNK forces you to dog-food on the tools and mindset to understand developers better, and as a designer or developer — to learn React Native <a href="http://learncodethehardway.org/"><em>the hard way</em></a> by doing, rather than by reading.</p> <p>RNK is an orchestrated series of drills, or katas, that deliberately but gently help you build your muscle memory, by letting you attempt to design and compose React Native views, and to fail and fix them as many times as it takes to win it.</p> <p>As a designer, you need to have a code editor with a simple npm setup (which you probably have if you’re doing front-end), and a basic grasp of files and folders (which again, you probably already have).</p> <p>As a developer, you should feel at home with what ever tools you already have.</p> <p><img src="https://rng0.io/blog/announcing-react-native-katas/1.png" alt="" /></p> <p>Up until now, I’m very happy with how RNK has been doing with designers, and I’ve been pleasantly surprised that even experienced developers like it (thanks <a href="https://medium.com/u/2c16a4556781">Yoni Weisbrod</a>)</p> <p><img src="https://rng0.io/blog/announcing-react-native-katas/2.png" alt="" /></p> <p>RNK is <a href="https://github.com/jondot/ReactNativeKatas">open source on Github</a>, and that means you can drive home your solutions and suggest improvements, as well as make up new and exciting Katas yourself by submitting a pull-request (that will be happily accepted!).</p> <p>Go ahead and start building your React Native chops!</p> <h2 id="one-last-thing">One Last Thing</h2> <p>Stickers!</p> <p><img src="https://rng0.io/blog/announcing-react-native-katas/3.png" alt="" /></p> <p>The React Native Kata Ninja!</p> <p>Send me an email: dotan/at/paracode.com and I’ll make sure these cross an ocean to get to you!</p> 10 Things you should know about React Native Mon, 20 Jun 2016 00:00:00 +0000 Unknown https://rng0.io/blog/10-things-you-should-know-about-react-native/ https://rng0.io/blog/10-things-you-should-know-about-react-native/ <p>If you’re just starting out, you probably have a ton of questions. Hopefully these pointers give you some confidence to go and try React Native today.</p> <h4 id="1-navigation-is-complex">1. Navigation is… complex</h4> <p>Read <a href="https://github.com/brentvatne/hard-react-native-problems/issues/12">this</a>, by yours truly.</p> <h2 id="2-redux-and-probably-most-javascript-libraries-you-know-work-flawlessly">2. Redux and probably most Javascript libraries you know work flawlessly</h2> <p>The surface area of React Native’s API is amazing. It stands out of your way when you want to adopt components from the React ecosystem such as <a href="https://github.com/reactjs/redux">Redux</a>, <a href="https://facebook.github.io/immutable-js/">Immutablejs</a> or vanilla Javascript libraries like <a href="https://lodash.com/">lodash</a> or <a href="http://momentjs.com">moment</a>.</p> <h2 id="3-you-don-t-break-when-ios-or-android-changes">3. You don’t break when iOS or Android changes</h2> <p>When you use a library that abstracts over the operating system SDK API, a peculiar thing happens to your codebase: it does not depend on that SDK directly. Surprisingly, this means the following:</p> <ul> <li>Less stress when facing major iOS releases</li> <li>Less breakage and deprecation of APIs</li> <li>No need to conform to new or changed language features (i.e. Swift)</li> <li>No need to adapt to new tools (i.e. Xcode, Intellij/Gradle)</li> </ul> <p>Effectively, you detach yourself from these concerns completely, and you can take defensive actions on your terms, in your time.</p> <h2 id="4-common-codebase-is-really-doable">4. Common codebase is really doable</h2> <p><a href="https://en.wikipedia.org/wiki/Write_once,_run_anywhere">Write once run everywhere?</a> I think history thought us that we all should give up on that one. But for React Native, when you call a component ProfileHeader, it can either be implemented commonly for iOS and Android, or separately, and your code still depends on that directly:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-cast z-expr z-ts"><span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">View</span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span> </span><span class="z-source z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">ProfileHeader</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span><span class="z-source z-ts"><span class="z-cast z-expr z-ts"> <span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">Settings</span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span> </span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">settings</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">map</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">k</span><span class="z-punctuation z-separator z-parameter z-ts">,</span><span class="z-variable z-parameter z-ts">v</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span></span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-cast z-expr z-ts"> <span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">View</span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-cast z-expr z-ts"> <span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">Label</span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">k</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">Label</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-cast z-expr z-ts"> <span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">Value</span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">v</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">Value</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">View</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span><span class="z-source z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">Settings</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span><span class="z-source z-ts"><span class="z-cast z-expr z-ts"> <span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">TabBar</span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span> </span><span class="z-source z-ts"><span class="z-cast z-expr z-ts"> <span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">Item</span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span><span class="z-variable z-other z-constant z-ts">A</span><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">Item</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span><span class="z-source z-ts"><span class="z-cast z-expr z-ts"> <span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">Item</span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span><span class="z-variable z-other z-constant z-ts">B</span><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">Item</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span><span class="z-source z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">TabBar</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span><span class="z-source z-ts"><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">View</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></code></pre> <p>And certainly things like TabBar will be specifically modeled per platform. It is <em>up to you</em> to abstract things in a way that your code will be reusable to as high degree as possible — that’s software engineering, and React Native simply uses React’s compositional model — which means you should be able to do what ever you want to.</p> <h2 id="5-css-without-the-suck">5. CSS without the suck</h2> <p>If you’re doing CSS on the Web, this is what you’ll be looking at, on every project (multiply that pain when you’re a team):</p> <ul> <li>Pick a higher-level language, such as Sass, Less, Stylus, to allow for variables, inheritance, and mixins in existing work</li> <li>Set up a build pipeline for that</li> <li>Go through a browser-specific prefixing processor to add all the pesky -moz, -webkit etc. prefixes.</li> <li>Minify all that</li> <li>Fight browser inconsistency for layout and style</li> </ul> <p>React Native picks Javascript as the styling language. With this you’re getting variables, inheritance, mixins, and really anything you might imagine. And, since that’s code, you don’t need to minify it.</p> <p>Layout is done solely with React Native’s implementation of flexbox, which is arguably one of the most successful layout models a developer can use today. Everything else that exists in CSS was left out, so I say: this is “CSS — the good parts”.</p> <p>You can dive right into React Native styling without prior knowledge, with <img src="https://rng0.io/blog/10-things-you-should-know-about-react-native/3.png" alt="React Native Katas" /></p> <h4 id="6-use-any-language-that-eventually-compiles-to-javascript">6. Use any language that eventually compiles to javascript</h4> <p>Clojure is one beautiful language that I like. This means I’ll probably always be on the lookout for Clojurescript on React Native, and apparently, it has <a href="http://cljsrn.org/">relatively vibrant ecosystem</a>.</p> <p>So what ever flavor you choose as long as it compiles to Javascript is valid. Just remember, being a cutting edge technology, you’ll always get the best experience using what ever React Native already comes with.</p> <h2 id="7-avoid-javascript-fatigue">7. Avoid Javascript fatigue</h2> <p>Which brings me to this point: using React Native, you get to have a first-class modern Javascript experience without tiresome configuration of your build pipeline. Everywhere you’ll look, you’ll feel that developers come first; it’s already awesome by default.</p> <h2 id="8-remote-update-your-codebase-silently">8. Remote update your codebase, silently</h2> <p>Since your app is usually a slim native layer and a thick Javascript bundle, it is easy to be able to control how that Javascript bundle is maintained on and off the device’s storage.</p> <p>In your development workflow, things are already being served over the wire (that includes the Javascript bundle and hot updates), so React Native sort of falls into a use case for remote updates.</p> <p>Today, you can use <a href="https://microsoft.github.io/code-push/">CodePush</a> for free, in order to control remote updates to your app. Fix bugs, deploy new features and content without going through a tiresome appstore process (note that terms of service for Apple do state that you must not remote update features — but YMMV).</p> <h2 id="9-wrap-native-components-easily">9. Wrap native components easily</h2> <p>React Native comes with a bunch of macros on the iOS side and decorators on the Android side to allow you to expose native functionality declaratively. There’s nothing I’ve seen so far that makes it this easy (go take a look at <a href="https://github.com/apache/cordova-plugin-file-transfer">plugin for Cordova/PhoneGap</a>).</p> <p>Not only that, the entire React Native codebase is dogfooding on these building blocks, so you should feel confident that it works for everything you’ll try.</p> <h2 id="10-you-can-t-handle-binary-data-in-javascript-just-yet">10. You can’t handle binary data in Javascript just yet</h2> <p>React Native uses a bridge; a transport mechanism to pass messages such as “Call this native function with these parameters (Javascript to Native)” or “Here’s your event (Native to Javascript)” back and forth from Javascript and native domains.</p> <p>For simple and aggregate types, such as strings, ints, arrays and dictionaries, this works fine. However, for opaque types such as a binary buffer, it is less than optimal: currently there’s no support for passing binary data back and forth and apparently, it is because of a non-optimal Buffer implementation.</p> <p>The solution is one of two:</p> <ol> <li>Encode binary data as base64 and from there, you’re handling a string. This is how many, if not all of the component that pass typical binary data to the Javascript side work.</li> <li>Keep the original binary at the Native side, and build native wrapper modules to manipulate it through your own set of commands. The downside here is that you will have to do the book keeping in a reliable manner and ensure no binary piece of data is left stale on the device.</li> </ol> <ul> <li><a href="https://github.com/facebook/react-native/issues/2294">Support rendering image buffers</a></li> <li><a href="https://github.com/facebook/react-native/issues/5449">About Fetch API</a></li> <li><a href="https://github.com/facebook/react-native/issues/854">Does fetch with blob() marshal data across the bridge?</a></li> </ul> Routing and Navigation in React Native Sun, 19 Jun 2016 00:00:00 +0000 Unknown https://rng0.io/blog/routing-and-navigation-in-react-native/ https://rng0.io/blog/routing-and-navigation-in-react-native/ <p><img src="https://rng0.io/blog/routing-and-navigation-in-react-native/1.png" alt="" /></p> <p>Navigation is hard. Especially in mobile. In this article we’ll find out why, and how React Native was set to solve that.</p> <h2 id="why-navigation">Why navigation</h2> <p>You might ask yourself: why do you need navigation? What’s a navigation stack? Well, navigation on mobile apps covers these concerns:</p> <ul> <li>Situational awareness: know where you’re at, and possibly, where you came from</li> <li>Functional: be able to navigate. Go back, undo, skip, or “deep link” into some functionality of the app</li> <li>Infrastructural: consolidate ceremonies that need to be done before rendering a new screen.</li> <li>Maintainability: to realize the above concern, often, you’ll need to build a state machine. A navigation stack is a state machine. Platforms may let you code this state machine declaratively or visually, and promote maintainability by keeping all of that logic in one place.</li> </ul> <h2 id="why-navigation-is-scary">Why Navigation is Scary</h2> <p>In my opinion, one of the scariest topics in mobile application development is routing and navigation.</p> <p>Navigation infrastructure typically holds a navigation stack, which holds screens, which hold components and data. Often, this state is not represented anywhere; it is a transient state that the user have created by merely navigating around, which is why it makes it hard to reason about.</p> <p>Navigation flows are also hard to reproduce, should you bump into bugs in that area, and often these bugs carry memory leaks and the sorts (we did just mention a navigator stack holds reference to screens, which hold reference to components, which holds reference to data, and so on).</p> <p>Navigation UX deals with how each operating system brought with it its own truth as to how to properly do navigation, as well as how to facilitate it: Android’s implicit back button behavior, and iOS with its purely explicit text navigation bar.</p> <p>And lastly, each platform has its own tooling and how to ease up all said pains. Android has the traditional approach, where you build screens and code transitions with the omnipotent Intent, and lately iOS offered a more intuitive approach with Storyboards and almost physically connecting screens.</p> <h2 id="navigation-in-react-native">Navigation in React Native</h2> <p>The bad news is that navigation doesn’t get less scary to me, even with React Native. I guess that for me the saying “the more you know, the more you worry” painfully applies.</p> <p>However, the good news, or maybe the great news is — that with the React, Javascript, and React Native combo everything becomes amazingly more predictable.</p> <ul> <li>React sports one-way binding, which helps us reason about our code. So to find memory leaks we travel a one-directional dependency tree (YMMV).</li> <li>Javascript brings about the Chrome Devtools, which by now are one of the most powerful toolsets for developers. If you’re going to build something complex, you’re lucky to be living in 2015 (at least, that’s when this book was written, consider yourself luckier if you’re from the future!).</li> <li>React Native gives us plenty of escape hatches. This is the case where we want to use NavigatorIOS, the native iOS navigation stack, in order to minimize the element of surprise by not emulating a navigation stack, or simply put — sticking to the host OS best practices.</li> </ul> <p>In addition, if what you’re building is really painfully complex (years of effort and complexity), I guess at this stage you probably don’t really need this book and should build your app on each platform separately using dedicated tooling and approaches, just to be safe.</p> <p>Let’s take a look at the two React Native navigation solutions.</p> <h2 id="navigator-vs-navigatorios">Navigator vs. NavigatorIOS</h2> <p>React Native started on iOS, and this is why the best-in-class or perhaps most used navigation stack is NavigatorIOS. If you’re building an iOS only app and want to play it safe, use it. However, if a year or more passed from the writing of the book, it might be possible that the generic Navigator dwarfed NavigatorIOS in features and stability, so allow yourself to evaluate that again by Googling around.</p> <p>Navigator is React Native’s attempt to abstract the concept of navigation. At the time of writing, it is good-but-not-great a bet for a smooth ride if you want to keep a common navigation codebase for iOS and Android.</p> <p>We’ll use NavigatorIOS for iOS and Navigator for Android. Even if reality changes and Navigator becomes the be-all-end-all navigation solution for both platforms, I would want to keep it apart. It feels that navigation on iOS and Android will always want to be different, somehow, either UX or hardware-wise, so it makes sense to future-proof this and make room for code to evolve differently.</p> <h2 id="navigator">Navigator</h2> <p>Let’s go through a brief Navigator example. For this, we’ll assume the common master-detail pattern, where we have a master view containing a list of items, and then when tapping an item we want to navigate to a child (or detail) item.</p> <p><img src="https://rng0.io/blog/routing-and-navigation-in-react-native/2.png" alt="" /></p> <h2 id="wiring-navigator">Wiring Navigator</h2> <p>This is what you probably wanted to read before I filled your head with back button craze.</p> <p>The Navigator is a React component that deals with two main concerns:</p> <ol> <li>Returning a properly configured Navigator component, with icon, back icon, colors, initial route and more. In addition, it should wire a renderScene handler, which we talk about next.</li> <li>A renderScene handler implementation, which is a function that takes care of mapping a route (which just a Javascript object) to a screen of our choosing. The goal is similar in concept to backend routing with frameworks like Express or Rails, which deals with separating routing from destination.</li> </ol> <p>Here is how we set that up, for a simple two-screen (“first” to “second”) navigation flow:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-storage z-type z-class z-ts">class</span> <span class="z-entity z-name z-type z-class z-ts">Navigation</span> <span class="z-storage z-modifier z-ts">extends</span> <span class="z-entity z-name z-type z-module z-ts">React</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-other z-inherited-class z-ts">Component</span><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">render</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-support z-class z-dom z-ts">Navigator</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">style</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>styles.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">container</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">initialRoute</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>{<span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">id</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>first<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span>} </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">renderScene</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>this.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">navigatorRenderScene</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">navigatorRenderScene</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">route</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">navigator</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">_navigator</span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-support z-variable z-dom z-ts">navigator</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-switch-statement z-expr z-ts"><span class="z-switch-expression z-expr z-ts"><span class="z-keyword z-control z-switch z-ts">switch</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-object z-ts">route</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">id</span><span class="z-meta z-brace z-round z-ts">)</span></span> <span class="z-switch-block z-expr z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-switch-statement z-expr z-ts"><span class="z-switch-block z-expr z-ts"> <span class="z-case-clause z-expr z-ts"><span class="z-keyword z-control z-switch z-ts">case</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>first<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-case-clause z-expr z-ts"><span class="z-punctuation z-definition z-section z-case-statement z-ts">:</span></span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-switch-statement z-expr z-ts"><span class="z-switch-block z-expr z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-cast z-expr z-ts"><span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">First</span> <span class="z-entity z-name z-type z-ts">navigator</span>=<span class="z-meta z-object z-type z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-field z-declaration z-ts"><span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">navigator</span></span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-entity z-name z-type z-ts">title</span>=<span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>first<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span>/<span class="z-meta z-brace z-angle z-ts">&gt;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-switch-statement z-expr z-ts"><span class="z-switch-block z-expr z-ts"> <span class="z-case-clause z-expr z-ts"><span class="z-keyword z-control z-switch z-ts">case</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>second<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-case-clause z-expr z-ts"><span class="z-punctuation z-definition z-section z-case-statement z-ts">:</span></span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-switch-statement z-expr z-ts"><span class="z-switch-block z-expr z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-cast z-expr z-ts"><span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">Second</span> <span class="z-entity z-name z-type z-ts">navigator</span>=<span class="z-meta z-object z-type z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-field z-declaration z-ts"><span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">navigator</span></span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-entity z-name z-type z-ts">title</span>=<span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>second<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span> /<span class="z-meta z-brace z-angle z-ts">&gt;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-switch-statement z-expr z-ts"><span class="z-switch-block z-expr z-ts"> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p>In the render function, we set up a Navigator, in which the most important properties to specify are the initialRoute and renderScene. Note that a <em>route</em> is simply any Javascript object that will be carried from screen to screen. Here, we make a convention to have our main routing concern exist behind an id property; if we have anything else to pass, we’ll use a suitable payload, within the same object. However, let’s agree that id will always be reserved for routing.</p> <p>The good news is that we passed the clunky part of declaring a Navigator component and then wiring up screens. It <em>can</em> become less clunky, maybe, some day. It <em>can</em> have some sort of a DSLish feel to it such as:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> This doesn&#39;t really exist, but you can make it</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">var</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">Navigation</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">routes</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>first<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">route</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">navigator</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span></span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">First</span> <span class="z-variable z-other z-readwrite z-ts">title</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>foobar<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span></span> </span></code></pre> <p>But this kind of experience doesn’t exist yet. It’s quite a low-hanging fruit, so think of it as a nice weekend project to do :)</p> <p>Next up, our screens are really normal views that take care of massaging a ToolbarAndroid to their needs. A ToolbarAndroid is our top navigation bar, and with React Native, it is quite flexible. So flexible, in fact, that we need to code every decision point such as when to show the back icon, what kind of title to display and so on, based on our current and past screens.</p> <p>A ToolbarAndroid is also the real Android Toolbar widget (see <a href="https://developer.android.com/intl/zh-tw/reference/android/support/v7/widget/Toolbar.html">here</a>) which, in accordance to our theme here (showing the platform-specific way first) is an Android-specific component.</p> <blockquote> <p>Flexibility can be good and bad. Good since on each screen, we can specify exactly how we want things to be had on the navbar. Bad, because we need to be defensive here since it can become a maintenance overhead or a state machine of the worst kind — the one that is spread out through out our entire codebase.</p> </blockquote> <p>Here is our Second screen:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-storage z-type z-class z-ts">class</span> <span class="z-entity z-name z-type z-class z-ts">Second</span> <span class="z-storage z-modifier z-ts">extends</span> <span class="z-entity z-name z-type z-module z-ts">React</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-other z-inherited-class z-ts">Component</span><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">render</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">View</span> <span class="z-variable z-other z-readwrite z-ts">style</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>styles.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">container</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">ToolbarAndroid</span> <span class="z-variable z-other z-readwrite z-ts">style</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>styles.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">toolbar</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">title</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>this.props.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">title</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">navIcon</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">require</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>image!ic_arrow_back_white_24dp<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">onIconClicked</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>this.props.navigator.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">pop</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">titleColor</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>#FFFFFF<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-cast z-expr z-ts"> <span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">Text</span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">Second</span> <span class="z-support z-variable z-dom z-ts">screen</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-support z-class z-dom z-ts">Text</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">View</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <p>If you’re trying this out, make sure that ic_arrow_back_white_24dp is an icon you’ve dropped in your resources folder — in this case Android. For the sake of the experiment you can use a single hi-res image for all size variants.</p> <p>Note here, that we specify our navIcon explicitly. We want users to be able to tap a back icon right on the navbar.</p> <p>Next up, our First screen:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-storage z-type z-class z-ts">class</span> <span class="z-entity z-name z-type z-class z-ts">First</span> <span class="z-storage z-modifier z-ts">extends</span> <span class="z-entity z-name z-type z-module z-ts">React</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-other z-inherited-class z-ts">Component</span><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">navSecond</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">props</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">navigator</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">push</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">id</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>second<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">render</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">View</span> <span class="z-variable z-other z-readwrite z-ts">style</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>styles.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">container</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">ToolbarAndroid</span> <span class="z-variable z-other z-readwrite z-ts">style</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>styles.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">toolbar</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">title</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>this.props.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">title</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">titleColor</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>#FFFFFF<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">TouchableHighlight</span> <span class="z-variable z-other z-readwrite z-ts">onPress</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>this.navSecond.bind(this)<span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-cast z-expr z-ts"> <span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">Text</span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span><span class="z-variable z-other z-readwrite z-ts">Navigate</span> <span class="z-variable z-other z-readwrite z-ts">to</span> <span class="z-variable z-other z-readwrite z-ts">second</span> <span class="z-support z-variable z-dom z-ts">screen</span><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-support z-class z-dom z-ts">Text</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">TouchableHighlight</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">View</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p>Here, we don’t specify a back button, because we recognize it as the root screen. The most important (and fun) piece of code here is how we navigate to the Second screen. We just push a new object which happens to contain an id that we agreed is our routing property that we base our routing on. Note that we don’t specify the screen type, object, or tag here — this is the essence of separating routing from destination or implementation, which is a Good Thing.</p> <p>This completes our vanilla Navigator walkthrough. Next up we’ll take a look at its older brother: NavigatorIOS.</p> <p>On Android, we’ll have to take care of navigation, and also the back button.</p> <h2 id="android-s-back-button">Android’s Back Button</h2> <p><img src="https://rng0.io/blog/routing-and-navigation-in-react-native/3.png" alt="" /></p> <p>Tapping into the back button is unique to Android devices, in that it is on occasion a “hardware” button on a dedicated touch area of the glass as in Samsung phones or HTC phones, or a software button that renders at the lower part of the screen (common on the Nexus family of devices).</p> <p>The following boilerplate is more or less needed verbatim in our app to support responding to the back button. Note that the _navigator variable is scope-global, and it gets filled on first navigation. First read this snippet of code to understand what’s going on, and then I’d recommend tracking the _navigator variable throughout as well.</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">var</span> <span class="z-meta z-object-binding-pattern-variable z-ts"><span class="z-punctuation z-definition z-binding-pattern z-object z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-object-binding-pattern-variable z-ts"> ...<span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-object-binding-pattern-variable z-ts"> ...<span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-object-binding-pattern-variable z-ts"> ...<span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-object-binding-pattern-variable z-ts"> <span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">BackAndroid</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-object-binding-pattern-variable z-ts"><span class="z-punctuation z-definition z-binding-pattern z-object z-ts">}</span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-readwrite z-ts">React</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">var</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">_navigator</span></span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> we fill this up upon on first navigation.</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">BackAndroid</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">addEventListener</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>hardwareBackPress<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">_navigator</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">getCurrentRoutes</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span> <span class="z-keyword z-operator z-comparison z-ts">===</span> <span class="z-constant z-numeric z-decimal z-ts">1</span> <span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-constant z-language z-boolean z-false z-ts">false</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">_navigator</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">pop</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-constant z-language z-boolean z-true z-ts">true</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <p>BackAndroid is a simple library that binds to the native events of the host device.</p> <p><em>Deep dive alert!</em>: here’s how it works (notice the special exitApp case):</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">RCTDeviceEventEmitter</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">addListener</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-constant z-ts">DEVICE_BACK_EVENT</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-function z-expression z-ts"><span class="z-storage z-type z-function z-ts">function</span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">var</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">invokeDefault</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-language z-boolean z-true z-ts">true</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">_backPressSubscriptions</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">forEach</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">subscription</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">subscription</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">invokeDefault</span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-language z-boolean z-false z-ts">false</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">invokeDefault</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">BackAndroid</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">exitApp</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <p>On iOS you need not worry if you’re mistakenly (or if you lazily want to use the exact same code) using the BackAndroid module — all of the functions it carries are no-ops.</p> <blockquote> <p>As with any event listener, when you add a listener, you must immediately ask yourself how you are going to remove it — does the subscribe method return a special handle you need to provide when canceling?, or do you have to do the bookkeeping yourself and come up with the same reference to the handler you provided?</p> </blockquote> <p>In the BackAndroid case, when we subscribe we must keep a reference to the handler function we give it. However, do we really see ourselves disabling the back button in real life?</p> <h2 id="toolbarandroid-and-navigator-navigationbar">ToolbarAndroid and Navigator.NavigationBar</h2> <p>It is important to notice that Navigator comes with a pre-baked piece of UI that functions as your top Android navigation bar. If you’ll dig into <a href="https://github.com/facebook/react-native/tree/master/Examples/UIExplorer/Navigator">UIExplorer</a> within the facebook/react-native Github repo, you’ll see these kinds of things:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">var</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">NavigationBarRouteMapper</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-entity z-name z-function z-ts">LeftButton</span></span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-meta z-function z-expression z-ts"><span class="z-storage z-type z-function z-ts">function</span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">route</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">navigator</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">index</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">navState</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">index</span> <span class="z-keyword z-operator z-comparison z-ts">===</span> <span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-constant z-language z-null z-ts">null</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">var</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">previousRoute</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-object z-ts">navState</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">routeStack</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">index</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-constant z-numeric z-decimal z-ts">1</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-spread z-ts">...</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-entity z-name z-function z-ts">RightButton</span></span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-meta z-function z-expression z-ts"><span class="z-storage z-type z-function z-ts">function</span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">route</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">navigator</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">index</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">navState</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-spread z-ts">...</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-entity z-name z-function z-ts">Title</span></span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-meta z-function z-expression z-ts"><span class="z-storage z-type z-function z-ts">function</span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">route</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">navigator</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">index</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">navState</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-spread z-ts">...</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"><span class="z-meta z-function z-expression z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <p>That is a concept of a NavigationBarRouteMapper or in other words, you’re telling your Navigator how to compose the NavigationBar’s UI on each and every route.</p> <p>Then, you stick an actual NavigationBar into your Navigator this way:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-support z-class z-dom z-ts">Navigator</span> </span><span class="z-source z-ts"> <span class="z-keyword z-operator z-spread z-ts">...</span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">navigationBar</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-objectliteral z-ts"> &lt;Navigator.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">NavigationBar</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">routeMapper</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">NavigationBarRouteMapper</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-objectliteral z-ts"> /&gt; </span></span><span class="z-source z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p>So what you’ve got here, is a Navigator which we learned how to configure and tell it how to render itself as a response to a given route, and a NavigationBar that pretty much does the same.</p> <blockquote> <p>Immediately, you start thinking — why separate the two? Your NavigationBar and Navigator rendering is spread out through two different mappers, and also, why not make a route a first-class citizen and making itself render? something like ShowCartRoute.render(), and then ShowCartRoute.renderBar()? well, this is what Exponentjs/ex-navigator solves exactly, and more or less feels like. I recommend <a href="https://github.com/exponentjs/ex-navigator">checking it out</a>. Also, you might imagine that this is not going to be the only flavor people would like to do their routing with, so keep watching out for new things.</p> </blockquote> <p>Should you use a ToolbarAndroid or a Navigator’s own NavigationBar? The answer is again — tradeoff. The NavigationBar is strongly tied to routes, ToolbarAndroid is tied to your view. ToolbarAndroid is, well, Android, and you could probably implement such a thing yourself generically.</p> <p>So bottom line, if you’re implementing something simple, then go with iOS’s NavigatorIOS for iOS (coming in the next section), and ToolbarAndroid for Android. Otherwise, use a Navigator for both, and either NavigationBar, or your own piece of bar that you construct manually. And, of course, do a quick browse on my <a href="https://github.com/jondot/awesome-react-native">awesome-react-native</a> list for any fancier bar that you’d like.</p> <h2 id="navigatorios">NavigatorIOS</h2> <p>This navigator is a bit simpler, and relies on the native iOS navigation stack. The iOS navigation stack is a powerful beast, it is only encouraging that it is hidden under such a simple React Native component, however be sure that if we wanted to do more involved things (custom Segues and such) it might have become trickier.</p> <p>For now, let’s implement the same Navigator example with NavigatorIOS.</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-storage z-type z-class z-ts">class</span> <span class="z-entity z-name z-type z-class z-ts">Navigation</span> <span class="z-storage z-modifier z-ts">extends</span> <span class="z-entity z-name z-type z-module z-ts">React</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-other z-inherited-class z-ts">Component</span><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">render</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-parameter z-ts">NavigatorIOS</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">style</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>styles.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">container</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">initialRoute</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>{ </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">title</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>first<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">component</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-readwrite z-ts">First</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span>} <span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p>Our Navigation component is simple and explicit, and the initial route specifies the verbatim component (here First) that we want to run. As a side note, you might also like to call this component Router or Handler or anything that represents a concept of a routing shell component.</p> <p>Next, we take a look at both our screens, First and Second.</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-storage z-type z-class z-ts">class</span> <span class="z-entity z-name z-type z-class z-ts">First</span> <span class="z-storage z-modifier z-ts">extends</span> <span class="z-entity z-name z-type z-module z-ts">React</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-other z-inherited-class z-ts">Component</span><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">navSecond</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">props</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">navigator</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">push</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">title</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>second<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">component</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-readwrite z-ts">Second</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">render</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">View</span> <span class="z-variable z-other z-readwrite z-ts">style</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>styles.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">content</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">TouchableHighlight</span> <span class="z-variable z-other z-readwrite z-ts">onPress</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>this.navSecond.bind(this)<span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-cast z-expr z-ts"> <span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">Text</span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span><span class="z-variable z-other z-readwrite z-ts">Navigate</span> <span class="z-variable z-other z-readwrite z-ts">to</span> <span class="z-variable z-other z-readwrite z-ts">second</span> <span class="z-support z-variable z-dom z-ts">screen</span><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-support z-class z-dom z-ts">Text</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">TouchableHighlight</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">View</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span></code></pre> <p>Somewhat similar in structure to Navigator, however again we see explicit mention of the Second screen.</p> <blockquote> <p>Note that when we use a NavigatorIOS and a plain child View as in this case, we will need to handle the height of a typical iOS navigation bar. In other words, we need to add a paddingTop property to our View such that the content will be offset from under the navigation bar. On more advanced components, such as the ScrollView, we might want to look for the automaticallyAdjustContentInsets property that allows the component to handle this for us automatically.</p> </blockquote> <p>The Second screen looks at least as simple:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-storage z-type z-class z-ts">class</span> <span class="z-entity z-name z-type z-class z-ts">Second</span> <span class="z-storage z-modifier z-ts">extends</span> <span class="z-entity z-name z-type z-module z-ts">React</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-other z-inherited-class z-ts">Component</span><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">render</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">View</span> <span class="z-variable z-other z-readwrite z-ts">style</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>styles.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">container</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-cast z-expr z-ts"> <span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">Text</span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">Second</span> <span class="z-support z-variable z-dom z-ts">screen</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-support z-class z-dom z-ts">Text</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">View</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <p>As mentioned during the Navigator overview, NavigatorIOS is simpler — we don’t have any navigation bar to tweak here, and in this case the view is completely vanilla — reusable and clean as-is.</p> <h2 id="passing-data">Passing Data</h2> <p>Often when doing navigation, in addition to the route, we want to pass an additional data, for the new screen-to-be to root on.</p> <p>In the generic Navigator component, data is passed as part of the plain Javascript object (in this example, the data property), so we can do something like this:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-switch-statement z-expr z-ts"><span class="z-switch-expression z-expr z-ts"><span class="z-keyword z-control z-switch z-ts">switch</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-object z-ts">route</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">id</span><span class="z-meta z-brace z-round z-ts">)</span></span> <span class="z-switch-block z-expr z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-switch-statement z-expr z-ts"><span class="z-switch-block z-expr z-ts"> <span class="z-case-clause z-expr z-ts"><span class="z-keyword z-control z-switch z-ts">case</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>first<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-case-clause z-expr z-ts"><span class="z-punctuation z-definition z-section z-case-statement z-ts">:</span></span> </span></span></span><span class="z-source z-ts"><span class="z-switch-statement z-expr z-ts"><span class="z-switch-block z-expr z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-cast z-expr z-ts"><span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">First</span> <span class="z-entity z-name z-type z-ts">navigator</span>=<span class="z-meta z-object z-type z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-field z-declaration z-ts"><span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">navigator</span></span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-entity z-name z-type z-ts">data</span>=<span class="z-meta z-object z-type z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-entity z-name z-type z-module z-ts">route</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-meta z-field z-declaration z-ts"><span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">data</span></span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-entity z-name z-type z-ts">title</span>=<span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>first<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span>/<span class="z-meta z-brace z-angle z-ts">&gt;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></code></pre> <p>With NavigationBar it will mostly be the same, if you take care to pass your data within your routeMapper route object, and picking it apart in your routeMapper callbacks.</p> <p>With NavigatorIOS we need to use the special passProps property:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">NavigatorIOS</span> </span><span class="z-source z-ts"> <span class="z-variable z-other z-readwrite z-ts">initialRoute</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>{ </span></span><span class="z-source z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">component</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-readwrite z-ts">First</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">title</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>first<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">passProps</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">data</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>some-data<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span>} </span><span class="z-source z-ts"> <span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></code></pre> <p>And the receiving component will get both the data property and a special route and navigator props it can use to make decisions and to navigate further.</p> <blockquote> <p>There is a fascinating question about when a new screen is born due to a new navigation. Is the data that the screen just got from the route a pointer to data it needs to fetch?, or the actual data it needed to fetch verbatim?. No answer is wrong: it is a trade-off between being implicit and explicit, or defensive rather than naive, respectively.</p> </blockquote> <h2 id="patterns">Patterns</h2> <p><img src="https://rng0.io/blog/routing-and-navigation-in-react-native/4.png" alt="" /></p> <h2 id="search-in-navbar">Search in Navbar</h2> <ul> <li>NavigatorIOS — Can’t do this.</li> <li>Navigator, with Navigator.NavigationBar — make sure that your NavigationBar mapper will render your variant of a title like this:</li> </ul> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-entity z-name z-label z-ts">Title</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-meta z-function z-ts"><span class="z-storage z-type z-function z-ts">function</span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">route</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">navigator</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">index</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">navState</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">AwesomeSearchbar</span> <span class="z-keyword z-operator z-spread z-ts">...</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-brace z-round z-ts">)</span> </span></span></span></code></pre> <ul> <li>Navigator and ToolbarAndroid — make sure that the screen you are routing to, which is supposed to contain the search bar inlined within the navbar, should now render a <em>single child</em> which is your searchbar, prefer not providing a title in this case (styling and others are cut for brevity):</li> </ul> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-cast z-expr z-ts"><span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">ToolbarAndroid</span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span> </span><span class="z-source z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">AwesomeSearchbar</span> <span class="z-keyword z-operator z-spread z-ts">...</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span><span class="z-source z-ts"><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">ToolbarAndroid</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></code></pre> <p>A searchbar is interactive — it is completely OK to make the AwesomeSearchbar component interactive by supplying callbacks via props. You can pull these callbacks in two ways:</p> <ol> <li>NavigatorBar renderer — from your route, or by making the renderer take parameters.</li> <li>ToolbarAndroid — since it is rendered within a container view, simply the callbacks of the neareset smart component will do.</li> </ol> <h2 id="custom-content-in-title">Custom Content in Title</h2> <p>As in the previous pattern, the idea is similar sans the callbacks.</p> <ul> <li>NavigatorIOS — content is string only</li> <li>NavigatorBar and ToolbarAndroid — as with the previous pattern, simply hand over the component you want to render, this time no callbacks or interactivity needed.</li> </ul> <h2 id="routed-navbar-content">Routed Navbar Content</h2> <p>We saw this while handling NavigatorBar mapper — the content can change as a function of the route you are not traveling into.</p> <ul> <li>NavigatorIOS — each time you push a navigation, you can define how the coming-to-be navbar will look like, so this is a simple matter of using a new title:</li> </ul> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">props</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">navigator</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">push</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">title</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>some new title<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-keyword z-operator z-spread z-ts">...</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <ul> <li>Navigator and NavigationBar — use the route mapper, as we’ve seen before.</li> <li>ToolbarAndroid — this is trivial since you’re rendering it as part of the view, so you can couple the rendering to the view itself. Meaning, don’t fuss with the route, but with the actual view component content.</li> </ul> <h2 id="reactive-navbar-content">Reactive Navbar Content</h2> <p>There comes a time where your navbar changes, but not as a reaction to route changes, but to some kind of an external event, timed event, or anything reactive in nature.</p> <p>The way to solve it right now, which is gaining consensus, is to inject an event emitter to your app flows, and make sure that components on that navbar know to use it.</p> <ul> <li>NavigatorIOS — again, not possible</li> <li>Navigator.NavigationBar and ToolbarAndroid — make sure that the content you give each, will be able to use your global event emitter:</li> </ul> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-cast z-expr z-ts"><span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">ToolbarAndroid</span><span class="z-meta z-brace z-angle z-ts">&gt;</span></span> </span><span class="z-source z-ts"> <span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-variable z-other z-readwrite z-ts">AwesomeSearchbar</span> <span class="z-variable z-other z-readwrite z-ts">emitter</span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span>this.<span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">emitter</span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-operator z-spread z-ts">...</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span><span class="z-source z-ts"><span class="z-keyword z-operator z-relational z-ts">&lt;</span><span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-variable z-other z-readwrite z-ts">ToolbarAndroid</span><span class="z-keyword z-operator z-relational z-ts">&gt;</span> </span></code></pre> <blockquote> <p>You can also not use an explicit emitter but a Flux dispatcher, or do use an explicit emitter and inject it via something similar to React’s <a href="https://facebook.github.io/react/docs/context.html">context</a>, an so on. Since this is an advanced/thought material topic (under the “lightbulb” category) I’ll leave it to you for exploration.</p> </blockquote> <h2 id="getting-input">Getting Input</h2> <p>If you want to just go to a screen to collect input (i.e. modals), you can either use the actual Modal component (see <a href="https://facebook.github.io/react-native/docs/modal.html">here</a>), or send off a view with a callback within the renderScene logic block. At extreme cases you can use event emitter, or if you’re using Flux than a Flux action trivially solves it.</p> <h2 id="spreading-props">Spreading Props</h2> <p>If we scratch our head for a moment, we remember that routing in Navigator is done with a plain Javascript object. We can actually be opinionated and define this object as such:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">id</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>route-id<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">props</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">some</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>props<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p>And this way we can then do something like this, within our renderScene:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">navigatorRenderScene</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">route</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-support z-variable z-dom z-ts">navigator</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">_navigator</span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-support z-variable z-dom z-ts">navigator</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-cast z-expr z-ts"><span class="z-meta z-brace z-angle z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">First</span> <span class="z-entity z-name z-type z-ts">navigator</span>=<span class="z-meta z-object z-type z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-meta z-field z-declaration z-ts"><span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">navigator</span></span></span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-cast z-expr z-ts"> <span class="z-meta z-object z-type z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-keyword z-operator z-spread z-ts">...</span><span class="z-entity z-name z-type z-module z-ts">route</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-type z-ts">props</span><span class="z-punctuation z-definition z-block z-ts">}</span></span>/<span class="z-meta z-brace z-angle z-ts">&gt;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p>Using the new spread operator …, we easily inline the entire props bag of properties into our component. Remember React will go left-to-right on various props so you can enjoy a cascading effect (provide defaults and then override them with specific data)</p> <h2 id="summary">Summary</h2> <p>This concludes our discussion about routing. In summary, we’ve learned the following:</p> <ul> <li>Routing is hard, however at least on mobile, we have less rope to hang ourselves with by adopting the best practices of each mobile platform.</li> <li>React Native offers the generic and flexible Navigator for a general case use, and the older NavigatorIOS for iOS specific work. Choose the latter if you don’t need flexibility and want to run fast by getting the default iOS navigation stack behavior.</li> <li>Routing is either defined implicitly with Navigator and routes, or explicitly with NavigatorIOS and the specific components we route to.</li> <li>Remember to compensate for various UI glitches when using a navbar. Some content may vanish under the navbar because it is not padded.</li> <li>Passing data is easy through both Navigator and NavigatorIOS.</li> <li>NavigationBar is a common to both concept, it has its own tradeoffs.</li> </ul> Low Level Go Thu, 14 Jan 2016 00:00:00 +0000 Unknown https://rng0.io/blog/low-level-go/ https://rng0.io/blog/low-level-go/ <p>Let’s take a look at a golang binary size with real life dependencies.</p> <p>These are two of my own projects, where I knew they had to run on a command line, and across platforms:</p> <ul> <li><a href="https://github.com/jondot/castbox">Castbox</a> — 7.6Mb</li> <li><a href="https://github.com/jondot/groundcontrol">Ground Control</a> — 5.1Mb</li> </ul> <p>But what about when your work is very simple, and requires doing some classic C with direct operating system calls? Does it “pay off” to build that in Go?</p> <p>Some would write a small C program and be done with it; being small it would probably be a lot of fun since there’s no abstraction that can stop you anywhere.</p> <p>However, a one-off C program will <em>probably</em> compromise on portability and perhaps some other arguable properties such as code clarity and ease of maintenance. Let’s see what does it take to get as close as possible to that C program, but stay within Go.</p> <h2 id="reducing-the-go-binary-size">Reducing the Go Binary Size</h2> <p>Go binaries tend to grow fast with each included dependency. However, let’s make a very important statement: <strong>in real life, Go programs are small enough for that to not matter at all</strong>. Moreover, at runtime the resources consumed will be much less than say Java and Ruby and Python, which is perfect.</p> <p>That being said, what if we still want to get close to a C binary, and we don’t want to be in the MB range, but in KBs?</p> <h2 id="how-low-can-we-go">How low can we go?</h2> <p>The Go binary will pack the garbage collector, the goroutines scheduler and the dependencies you include via import.</p> <p>Taking a very minimal CLI program, let’s say we have os to cover basic file handling and flag to parse and access command line arguments. We don’t even do any common I/O operations here.</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-keyword z-other z-package z-go">package</span> <span class="z-variable z-other z-go">main</span> </span><span class="z-source z-go"><span class="z-keyword z-other z-import z-go">import</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span> </span><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>os<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>flag<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span><span class="z-source z-go"><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">main</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">os</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Open</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>foobar<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">flag</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Parse</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <p>Binary size: 1.8MB.</p> <p>Going foward, let’s remove flag and assume we can get to ARGV via os.Args. It’ll be less elegant but, whatever.</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-keyword z-other z-package z-go">package</span> <span class="z-variable z-other z-go">main</span> </span><span class="z-source z-go"><span class="z-keyword z-other z-import z-go">import</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span> </span><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>os<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span><span class="z-source z-go"><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">main</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">os</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Open</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>foobar<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <p>Binary size: 893Kb. That looks quite good. Can we do better?</p> <h2 id="syscall">syscall</h2> <p>Going through Go docs, we bump into <a href="http://golang.org/pkg/syscall">syscall</a>, Go’s interface into the low level OS primitives:</p> <blockquote> <p>The primary use of syscall is inside other packages that provide a more portable interface to the system, such as “os”, “time” and “net”. Use those packages rather than this one if you can.</p> </blockquote> <p>Let’s swap everything with “raw” syscalls:</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-keyword z-other z-package z-go">package</span> <span class="z-variable z-other z-go">main</span> </span><span class="z-source z-go"><span class="z-keyword z-other z-import z-go">import</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span> </span><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>syscall<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span><span class="z-source z-go"><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">main</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">syscall</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Open</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>foobar<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-constant z-numeric z-integer z-decimal z-go">0</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-constant z-numeric z-integer z-decimal z-go">666</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <p>Binary size: 544Kb. Neat (we’ll stop here — for anything practical, I assure you this <em>is</em> the bare minimum :).</p> <p>We can shave around 20KB more with <em>strip</em> but let’s forget about that for the moment.</p> <h2 id="shake-off-abstractions">Shake off abstractions</h2> <p>As with C, you can do without abstractions in Go. You could get a lot of mileage with syscall, as do a lot of the standard packages in order to implement the higher level, more streamlined Go API.</p> <p>Here is a snippet from <a href="https://github.com/jnwhiteh/golang/blob/master/src/os/file.go#L224">os.Chdir</a>, reassuring it’s a fancy wrapper around syscall.</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-meta z-function z-declaration z-go"><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-parameter z-go">f</span> <span class="z-keyword z-operator z-go">*</span><span class="z-storage z-type z-go">File</span><span class="z-punctuation z-section z-parens z-end z-go">)</span></span><span class="z-meta z-function z-declaration z-go"> <span class="z-entity z-name z-function z-go">Chdir</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">error</span></span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-other z-go">f</span> <span class="z-keyword z-operator z-go">==</span> <span class="z-constant z-language z-go">nil</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-variable z-other z-go">ErrInvalid</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-declaration z-go">e</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">syscall</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Fchdir</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">f</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">fd</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-terminator z-go">;</span> <span class="z-variable z-other z-go">e</span> <span class="z-keyword z-operator z-go">!=</span> <span class="z-constant z-language z-go">nil</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-keyword z-operator z-go">&amp;</span><span class="z-variable z-other z-go">PathError</span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>chdir<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">f</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">name</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">e</span><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-constant z-language z-go">nil</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <h2 id="summary">Summary</h2> <p>For a real life example, You can take a look at <a href="https://github.com/jondot/cronlock">cronlock</a>, a small utility I’ve built with the conclusions from this article. Like cron, it had to be small and simple.</p> <p>Working without abstractions from time to time is nice; it feels like being a kid with a LEGO again and there’s a special place for C in my heart to relay that feeling. Go makes that a tad bit more accessible and portable.</p> <p><em>Note:</em> syscall is supposed to be deprecated in favor of a better architecture, but has not yet — and the ideas here should still be valid after the transition. See more <a href="http://golang.org/s/go1.4-syscall">here</a>.</p> Ruby and Go Sitting in a Tree Sun, 03 Jan 2016 00:00:00 +0000 Unknown https://rng0.io/blog/ruby-and-go-sitting-in-a-tree/ https://rng0.io/blog/ruby-and-go-sitting-in-a-tree/ <p>In this post, we’ll see how to build a Ruby gem with Go based native extension <a href="https://github.com/jondot/scatter">I called scatter</a>, which is enabled by the latest Go 1.5 release and its c-shared build mode.</p> <p>Feel free to <a href="mailto:dotan@paracode.com">drop me a line</a> or <a href="https://twitter.com/jondot">tweet</a> if you have questions or comments.</p> <h2 id="c-shared-libraries-and-go-ruby-node-js-python">C Shared Libraries and Go, Ruby, Node.js, Python.</h2> <p>With Go 1.5, we’ve got a sparkling new go build -buildmode=c-shared mode, with which we can build C shared libraries.</p> <p>Go, Ruby, Node.js and Python (and others) use C shared libraries as extensions. As an example, if you wanted to parse XML and either your language didn’t have such a parser (as doubtful as it sounds), or wasn’t fast enough parsing, you could use the robust libxml C library which exists since almost forever (well, 99').</p> <p>The big deal is — Go 1.5 enables you to build libraries such as libxml, in Go instead of C, and then use them from any host language that can use C libraries.</p> <p>Here’s a crazy thing: <a href="http://id-rsa.pub/post/go15-calling-go-shared-libs-from-firefox-addon/">Firefox and Go</a> via a Firefox addon.</p> <h2 id="the-case-for-ruby-and-go">The Case for Ruby and Go</h2> <p>I don’t believe Ruby is slow for general purpose use cases, but it would be for specialized ones. When you’re using Ruby for the wrong workload, it might appear slow.</p> <p>With C extensions, and now Go — you don’t have to completely switch away from Ruby to tackle such a workload, when it doesn’t make sense.</p> <h2 id="ruby-ffi-and-c-extensions">Ruby FFI and C Extensions</h2> <p><a href="https://en.wikipedia.org/wiki/Foreign_function_interface">FFI</a> should be <a href="https://github.com/ffi/ffi/wiki/Examples">very easy to use</a> in Ruby given that you already have a properly built C library.</p> <p>Usually, you would use FFI on an existing, well known library and create what is called a “binding”, much like any of <a href="https://github.com/ffi/ffi/wiki/Projects-Using-FFI">these projects</a>.</p> <p>So to make Ruby call C code, for performance or simply for standing on <a href="https://en.wikiquote.org/wiki/Isaac_Newton">shoulders of giants</a> we would:</p> <ul> <li>Build a C library and export functions for FFI (which is really just using existing functions), or,</li> <li>Find an existing C library and point FFI at its API</li> </ul> <p>And of course with Go 1.5,</p> <ul> <li>Build a Go library and expose it as a C shared library, and point FFI at its API</li> </ul> <h2 id="getting-started">Getting Started</h2> <p>Building a C shared library based on Go is easy, here’s a simple example. The Go part:</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-comment z-line z-go"><span class="z-punctuation z-definition z-comment z-go">//</span> libsq.go </span></span><span class="z-source z-go"><span class="z-keyword z-other z-package z-go">package</span> <span class="z-variable z-other z-go">main</span> </span><span class="z-source z-go"><span class="z-keyword z-other z-import z-go">import</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>C<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span><span class="z-source z-go"> </span><span class="z-source z-go"><span class="z-comment z-line z-go"><span class="z-punctuation z-definition z-comment z-go">//</span>export sq </span></span><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">sq</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-parameter z-go">num</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">int</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">int</span></span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-variable z-other z-go">num</span><span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">num</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span><span class="z-source z-go"> </span><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">main</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <p>Here, the export comment is meaningful and Go uses it to mark out functions to export. Building as a C shared library goes like this:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> go build<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>buildmode</span><span class="z-keyword z-operator z-assignment z-option z-shell">=</span>c-shared<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>o</span> libsq.so libsq.go</span> </span></code></pre> <p>And now we can tie this up with Ruby and FFI. The Ruby part:</p> <pre data-lang="ruby" class="language-ruby z-code"><code class="language-ruby" data-lang="ruby"><span class="z-source z-ruby"><span class="z-comment z-line z-number-sign z-ruby"><span class="z-punctuation z-definition z-comment z-ruby">#</span> sq.rb </span></span><span class="z-source z-ruby"><span class="z-meta z-require z-ruby"><span class="z-keyword z-other z-special-method z-ruby">require</span> <span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-single z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&#39;</span>ffi<span class="z-punctuation z-definition z-string z-end z-ruby">&#39;</span></span></span></span> </span><span class="z-source z-ruby"><span class="z-meta z-module z-ruby"><span class="z-keyword z-control z-module z-ruby">module</span></span><span class="z-meta z-module z-ruby"> </span><span class="z-meta z-module z-ruby"><span class="z-entity z-name z-module z-ruby">Sq</span></span> </span><span class="z-source z-ruby"> <span class="z-keyword z-other z-special-method z-ruby">extend</span> <span class="z-support z-class z-ruby">FFI</span><span class="z-punctuation z-accessor z-double-colon z-ruby">::</span>Library </span><span class="z-source z-ruby"> ffi_lib <span class="z-support z-class z-ruby">File</span><span class="z-punctuation z-accessor z-dot z-ruby">.</span>expand_path<span class="z-punctuation z-definition z-group z-begin z-ruby">(</span><span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-double z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&quot;</span>./libsq.so<span class="z-punctuation z-definition z-string z-end z-ruby">&quot;</span></span></span><span class="z-punctuation z-separator z-sequence z-ruby">,</span> <span class="z-support z-class z-ruby">File</span><span class="z-punctuation z-accessor z-dot z-ruby">.</span>dirname<span class="z-punctuation z-definition z-group z-begin z-ruby">(</span><span class="z-variable z-language z-ruby">__FILE__</span><span class="z-punctuation z-definition z-group z-end z-ruby">)</span><span class="z-punctuation z-definition z-group z-end z-ruby">)</span> </span><span class="z-source z-ruby"> attach_function <span class="z-constant z-other z-symbol z-ruby"><span class="z-punctuation z-definition z-constant z-ruby">:</span>sq</span><span class="z-punctuation z-separator z-sequence z-ruby">,</span> <span class="z-punctuation z-section z-array z-ruby">[</span><span class="z-constant z-other z-symbol z-ruby"><span class="z-punctuation z-definition z-constant z-ruby">:</span>int</span><span class="z-punctuation z-section z-array z-ruby">]</span><span class="z-punctuation z-separator z-sequence z-ruby">,</span> <span class="z-constant z-other z-symbol z-ruby"><span class="z-punctuation z-definition z-constant z-ruby">:</span>int</span> </span><span class="z-source z-ruby"><span class="z-keyword z-control z-ruby">end</span> </span><span class="z-source z-ruby"> </span><span class="z-source z-ruby"><span class="z-comment z-line z-number-sign z-ruby"><span class="z-punctuation z-definition z-comment z-ruby">#</span> test it out </span></span><span class="z-source z-ruby"><span class="z-support z-class z-ruby">Sq</span><span class="z-punctuation z-accessor z-dot z-ruby">.</span>sq<span class="z-punctuation z-definition z-group z-begin z-ruby">(</span><span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-double z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&quot;</span>foobar<span class="z-punctuation z-definition z-string z-end z-ruby">&quot;</span></span></span><span class="z-punctuation z-definition z-group z-end z-ruby">)</span> </span></code></pre> <p>We’re done with the basic example. Seeing how easy this was, you probably are starting to have a ton of ideas. Note them down.</p> <p>But before you get started, let’s take a look at an expanded, more practical, example.</p> <h2 id="scatter-go-powered-parallel-http-requests-ruby-gem">Scatter — Go Powered Parallel HTTP Requests Ruby Gem</h2> <p>Scatter is a <a href="https://github.com/jondot/scatter">showcase gem</a> that does a scatter/gather (or fan-out, etc) type requests on a list of URLs in parallel.</p> <p>For somewhat pragmatic purposes (there <em>are</em> other <a href="https://github.com/typhoeus/typhoeus">C based gems</a> that do this) I’d like this kind of heavy lifting to be performed by Go’s <a href="https://golang.org/pkg/net/http">HTTP stack</a>, and modeled with Go’s <a href="http://talks.golang.org/2012/concurrency.slide">channels and goroutines</a> for concurrency.</p> <h2 id="libscatter-go">libscatter.go</h2> <p>We start, as before, by building our Go based C shared library. We’ll make a single function that coordinates all of the HTTP requests.</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-comment z-line z-go"><span class="z-punctuation z-definition z-comment z-go">//</span> somewhere in func scatter_request... </span></span><span class="z-source z-go"><span class="z-keyword z-control z-go">for</span> <span class="z-variable z-language z-blank z-go">_</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">uri</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-keyword z-other z-go">range</span> <span class="z-variable z-other z-go">cmd</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">URIs</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">go</span> <span class="z-storage z-type z-keyword z-function z-go">func</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-parameter z-go">_uri</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">c</span> <span class="z-keyword z-operator z-go">&lt;-</span> <span class="z-variable z-function z-go">makeRequest</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">_uri</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">uri</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span><span class="z-source z-go"> </span><span class="z-source z-go"><span class="z-variable z-declaration z-go">result</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">Result</span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span><span class="z-source z-go"><span class="z-keyword z-control z-go">for</span> <span class="z-variable z-language z-blank z-go">_</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-keyword z-other z-go">range</span> <span class="z-variable z-other z-go">cmd</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">URIs</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">resp</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-keyword z-operator z-go">&lt;-</span><span class="z-variable z-other z-go">c</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">result</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-variable z-other z-go">resp</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">uri</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span> <span class="z-keyword z-operator z-assignment z-go">=</span> <span class="z-variable z-other z-go">resp</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <p>This is the core of the idea. From this, you see that there are no locks, no complex or dirty coordination to achieve concurrency.</p> <h2 id="pitfalls-and-patterns">Pitfalls and Patterns</h2> <p>In the simple sq example earlier before, I’ve used ints as the parameters and return values. This was intentional — it was hiding some pitfalls.</p> <p>You might want to pass <em>complex objects</em> down to Go from Ruby, arrays perhaps, and then you might want to return multiple values from Go back to Ruby like the idiomatic Go result and error.</p> <p>ruby-ffi solves the <a href="https://github.com/ffi/ffi/wiki/Structs">complex objects</a> case for you, but not always <a href="https://github.com/ffi/ffi/wiki/Pointers">arrays</a>, and when you want to return multiple parameters, you really should return a complex object (struct).</p> <p>In this example, we want to:</p> <ul> <li>Pass multiple URLs, or even varargs</li> <li>Return a big result, which is an aggregation of all of the requests (array or hash)</li> <li>Signal an error if it happened</li> </ul> <p>Unfortunately we hit all of the mentioned points of doing FFI. We have to build structs manually, we have to limit array sizes to work with built in arrays, or we have to start dealing with pointers and unsafe references.</p> <h4 id="staying-happy">Staying Happy</h4> <p>I want to stay in my happy zone in terms of developer experience for now. This is why in this example we will defer parameter and return value coding to an application-side codec.</p> <p>We’ll pass a string in, and a string out. We’ll encode JSON in, and decode JSON out, and this could that easily be Protobuf, Thrift, msgpack, or Avro instead — what ever you like.</p> <p>Remember — this is a trick, but not a dirty one. It might very well take you where ever you want to go, without the headache.</p> <h2 id="working-with-strings">Working With Strings</h2> <p>So strings just became super important. And they should be — regardless of what we’re doing here strings were always the workhorse data type of software.</p> <p>But again, I used a deceptively simple example. I didn’t use strings intentionally, because if you tried simply doing this:</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-comment z-line z-go"><span class="z-punctuation z-definition z-comment z-go">//</span> libsq.go </span></span><span class="z-source z-go"><span class="z-keyword z-other z-package z-go">package</span> <span class="z-variable z-other z-go">main</span> </span><span class="z-source z-go"><span class="z-keyword z-other z-import z-go">import</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>C<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span><span class="z-source z-go"> </span><span class="z-source z-go"><span class="z-comment z-line z-go"><span class="z-punctuation z-definition z-comment z-go">//</span>export prnt </span></span><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">prnt</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-parameter z-go">str</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-function z-go"><span class="z-support z-function z-builtin z-go">println</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">str</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <p>It wouldn’t work.</p> <p>Although it looks like everything is working — str will be empty. This is because Ruby strings are illusively different than Go strings. To resolve it, we’ll do this:</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-comment z-line z-go"><span class="z-punctuation z-definition z-comment z-go">//</span> libsq.go </span></span><span class="z-source z-go"><span class="z-keyword z-other z-package z-go">package</span> <span class="z-variable z-other z-go">main</span> </span><span class="z-source z-go"><span class="z-keyword z-other z-import z-go">import</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>C<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span><span class="z-source z-go"> </span><span class="z-source z-go"><span class="z-comment z-line z-go"><span class="z-punctuation z-definition z-comment z-go">//</span>export prnt </span></span><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">prnt</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-parameter z-go">data</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">C</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">char</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-function z-go"><span class="z-support z-function z-builtin z-go">println</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">C</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">GoString</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">data</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <h2 id="gil">GIL</h2> <p>If your Go code works on a shared Ruby resource, you need to start juggling the <a href="https://en.wikipedia.org/wiki/Global_Interpreter_Lock">GIL</a>. This can be painful if you don’t have much experience with building robust concurrent code. And even if you do, concurrency is hard, and it’s harder when you also need to manage memory without a GC (in C extension land).</p> <p>So, if we stick to a somewhat naive principle where our Go extension does big things completely, we wouldn’t need to criss-cross between Ruby and Go code and maintain ownerships all around.</p> <h2 id="shared-memory">Shared Memory</h2> <p>You should read about <a href="http://blog.golang.org/c-go-cgo">Go and CGo memory management</a>, but hopefully, again, if we stick to the tips in this specific example — as harsh as they may sound, most of it would be irrelevant.</p> <h2 id="making-a-gem">Making a Gem</h2> <p>You now have all of the theory needed to build your own Go powered Ruby gems.</p> <p>Let’s take a look at the grunt work of laying out such a gem so that it would build a hybrid of Go and Ruby and run successfully after install for your users.</p> <h2 id="building-native-extensions">Building Native Extensions</h2> <p>Ruby gems support C Ruby and Java (JRuby) extensions by way of <a href="https://github.com/rake-compiler/rake-compiler">rake-compiler</a> It covers several concerns while building native extensions:</p> <ul> <li>Locating files and resources for building C or Java</li> <li>Generating a machine-specific context (makefiles, configuration) so that the build will address that specific machine’s architecture and bit width</li> <li>Running the build with the relevant tooling</li> <li>Installing and cleaning up</li> </ul> <p>What you need to do is:</p> <ul> <li>In your gemspec, point to an extconf.rb with spec.extensions = %w[ext/extconf.rb], your extconf file will be a simple and minimal descriptor of the build (out of our scope for this post)</li> </ul> <p>And, optionally:</p> <ul> <li>Provide instruction per environment (Java, C)</li> <li>Toggle inclusion of prebuilt binaries, that were cross-compiled ahead of time</li> </ul> <h2 id="building-our-native-go-extension">Building Our Native Go Extension</h2> <p>We don’t have a C codebase, nor do we need the C tooling. So we can do either of these two:</p> <ol> <li>Depend on Go tooling. A Go 1.5+ compiler should exist at the user’s machine, we’ll build the Go library on gem install time.</li> <li>Cross compile and include all binary artifacts with our gem. On install time, sniff out the platform and depend on the correct binary.</li> </ol> <p>We’ll do (1) because it feels cleaner for our purposes and makes debugging easier.</p> <p>If you’d like to streamline the workflow then I’d recommend (2). This way, the Ruby and Go codebase releases are allowed for independent (and healthy) evolution.</p> <h2 id="our-slim-extconf-and-makefile">Our Slim extconf and Makefile</h2> <p>Since Go makes a great toolset, this is how our extconf would look like :)</p> <pre data-lang="ruby" class="language-ruby z-code"><code class="language-ruby" data-lang="ruby"><span class="z-source z-ruby"><span class="z-comment z-line z-number-sign z-ruby"><span class="z-punctuation z-definition z-comment z-ruby">#</span> this is a cheat </span></span><span class="z-source z-ruby"><span class="z-support z-function z-builtin z-ruby">puts</span> <span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-double z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&quot;</span>running make<span class="z-punctuation z-definition z-string z-end z-ruby">&quot;</span></span></span> </span><span class="z-source z-ruby"><span class="z-meta z-string z-ruby"><span class="z-string z-interpolated z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">`</span>make build<span class="z-punctuation z-definition z-string z-end z-ruby">`</span></span></span> </span></code></pre> <p>And our makefile:</p> <pre data-lang="makefile" class="language-makefile z-code"><code class="language-makefile" data-lang="makefile"><span class="z-source z-makefile"><span class="z-meta z-function z-makefile"><span class="z-entity z-name z-function z-makefile">build</span></span><span class="z-keyword z-operator z-assignment z-makefile">:</span> <span class="z-meta z-function z-arguments z-makefile"></span><span class="z-meta z-function z-body z-makefile"></span></span><span class="z-source z-makefile"><span class="z-meta z-function z-body z-makefile"></span><span class="z-meta z-function z-body z-makefile"> <span class="z-source z-shell"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">go</span></span><span class="z-meta z-function-call z-arguments z-shell"> build<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>buildmode</span><span class="z-keyword z-operator z-assignment z-option z-shell">=</span>c-shared<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>o</span> libscatter.so libscatter.go</span></span> </span></span><span class="z-source z-makefile"><span class="z-meta z-function z-body z-makefile"></span><span class="z-comment z-line z-number-sign z-makefile"><span class="z-punctuation z-definition z-comment z-makefile">#</span> fake out clean and install </span></span><span class="z-source z-makefile"><span class="z-meta z-function z-makefile"><span class="z-entity z-name z-function z-makefile">clean</span></span><span class="z-keyword z-operator z-assignment z-makefile">:</span> <span class="z-meta z-function z-arguments z-makefile"></span><span class="z-meta z-function z-body z-makefile"></span></span><span class="z-source z-makefile"><span class="z-meta z-function z-body z-makefile"></span><span class="z-meta z-function z-makefile"><span class="z-entity z-name z-function z-makefile">install</span></span><span class="z-keyword z-operator z-assignment z-makefile">:</span> <span class="z-meta z-function z-arguments z-makefile"></span><span class="z-meta z-function z-body z-makefile"></span></span><span class="z-source z-makefile"><span class="z-meta z-function z-body z-makefile"></span> </span><span class="z-source z-makefile"><span class="z-meta z-function z-makefile"><span class="z-entity z-name z-function z-makefile">.PHONY</span></span><span class="z-keyword z-operator z-assignment z-makefile">:</span> <span class="z-meta z-function z-arguments z-makefile"><span class="z-string z-unquoted z-makefile">build</span></span><span class="z-meta z-function z-body z-makefile"> </span></span></code></pre> <p>That’s it. All the rest is standard Ruby gem, and our FFI glue code.</p> <h4 id="summing-up">Summing Up</h4> <p>This idea works well — you can clone the <a href="https://github.com/jondot/scatter">scatter repo</a>, install, run benchmarks and more.</p> <p>Personally, I fantasized about a Go powered Ruby gem ever seeing and trying to build a Rust based Gem, after reading <a href="http://blog.skylight.io/bending-the-curve-writing-safe-fast-native-gems-with-rust/">Rust and Skylight</a>.</p> <p>These kind of concoctions breath new life into a community and a platform.</p> <h2 id="make-sure-it-s-worth-it">Make Sure It’s Worth It</h2> <p>For the time of this writing, this idea is pretty much novel. There are not much official advise, and the road is paved with gotchas (again, see <a href="http://id-rsa.pub/post/go15-calling-go-shared-libs-from-firefox-addon/">Firefox and Go</a>).</p> <p>I think that for now, if you limit yourself to the what was presented here, you may be on the safe side.</p> <p>Truth be told, with Go, the friction for building native extensions goes down dramatically. Setting up such a Go extension is fun, rather than a full-fledged C extension. Suddenly “is it worth it?” becomes “why not?”.</p> <h2 id="the-real-world">The Real World</h2> <p>Ruby is fast enough for most things. However, you might find a Go based library useful for packing raw performance and cleaner concurrent codebase into:</p> <ul> <li>An existing project, that’s so old it wouldn’t respond well to an architecture refactor</li> <li>An existing team, that wouldn’t respond well for swapping an entire tech stack</li> <li>Ruby projects that miss libraries that Go already has, or has lower-quality libraries in the same sense.</li> </ul> 3 years of Go in Production Thu, 31 Dec 2015 00:00:00 +0000 Unknown https://rng0.io/blog/3-years-of-go-in-production/ https://rng0.io/blog/3-years-of-go-in-production/ <p>For the last 3 years, my microservices in production were divided into the following platforms:</p> <ul> <li><em>Core:</em> Ruby and JRuby with Rails and Sinatra</li> <li><em>Satellites, scale:</em> Node.js, Clojure and later: Go</li> </ul> <p>A “core” toolset would live long. It would also move fast. It would depict the domain of the business and the core product solution that provides raw value.</p> <p>Mostly, its performance profile doesn’t really introduce any infrastructural concerns. The satellites and “scale” toolset exhibits use cases where we bumped into scalability issues and had to tear apart a piece of the core, and rebuild it on top of a more performant stack.</p> <p>It also represent a pure infrastructural concern; such as a push infrastructure, or analytics services. These things don’t change as fast as the problem domain, and they do need to be robust, fast and dependable.</p> <p>I want to talk about that “scale” toolset and share a bit of my own experience. Let’s start at the end. While migrating from Node to Go, the following are the things I have noticed to be different.</p> <h2 id="crashes">Crashes</h2> <p>Node liked to crash when handling unexpected performance, mostly due to large use of connections and the overhead in managing them and keeping their resources in check. True, this is mostly solved by proper capacity planning, and usage of hardening patterns like <a href="http://martinfowler.com/bliki/CircuitBreaker.html">circuit breaker</a>, and throttling.</p> <p>However, using these, my Node services looked like forced concoctions that crashed hard and had a horrible GC profile of minutes per collection (this is early Node days I’m talking about where you had to patch V8 to support a larger heap). It kept reminding me V8 was originally built to be run on the desktop.</p> <p>Even in its early days, Go didn’t have any of that, and it was enough. And when it did crash, it recovered crazy fast, and I liked that property, and made use of it; failing processes <a href="http://c2.com/cgi/wiki?LetItCrash">crashed fast</a>.</p> <h2 id="performance">Performance</h2> <p>Around 2011–12, Node was at the apex of performance, and Go wasn’t. That is, until <a href="http://dave.cheney.net/2013/05/21/go-11-performance-improvements">Go 1.1</a> mixed up that equation. I first noticed it through the excellent <a href="https://www.techempower.com/benchmarks/">TechEm benchmarks</a>:</p> <ul> <li>Round 1, March 2013 (pre Go 1.1): Go at 13k req/s, Node at 10k req/s. No big deal.</li> <li>Round 10 (latest): Go at 380k req/s Node at 225k req/s. Around 175% increase in favor of Go, and when you compare that to Node with <a href="http://expressjs.com/">Express</a>, you get 145k req/s for Node, which is 260% increase in favor of Go.</li> </ul> <p>Although these are a merely but a specialized variant of a micro benchmark, think about the overhead of the web framework (Express) superimposed on the host platform (Node). When Go is straddled with a typical web framework (Gin), it doesn’t react that hysterically and the reduction in performance is in the 1–3% range, Node however had a dramatic reaction.</p> <p>It stands to hint which stack you’d want to pack infrastructure on. Think about it. Is this why a lot of full-on infrastructure projects ( <a href="https://docker.com">Docker</a>, <a href="https://github.com/kubernetes/kubernetes">Kubernetes</a>, <a href="https://www.vagrantup.com/">Vagrant</a> etc.) were built on Go (hint: yes)?</p> <h2 id="deployment">Deployment</h2> <p>To deploy Node or Ruby, you have to deal with dependencies. Tools such as <a href="http://bundler.io/">Bundler</a>, rubygems, and <a href="https://www.npmjs.com">npm</a> were created to overcome dependency hell, and provided us with an ever helpful layer of abstraction, which split our products into two:</p> <ul> <li>Product (essence)</li> <li>Product (dependencies)</li> </ul> <p>Essentially we could snapshot our deps, and ship our product. But notice, with Bundler and npm, we snapshot a <em>description</em> of our deps (unless we choose to vendor. IMHO, with npm — people usually don’t).</p> <p>Every deploy might have modified the dependency tree of the product, and servers hosting these products had to reflect that. For those wanting to solve this problem, they had to ask these questions:</p> <ul> <li>Is that the responsibility of the Configuration Management infra?</li> <li>Should the deployment process or framework take care of dependencies?</li> <li>Should you bundle dependencies along with your product?</li> <li>What happens when your dependencies die? (i.e. pulled off from Rubygems)</li> </ul> <p>And their answers would typically be:</p> <ul> <li>Configuration management should take care of the servers. Resources are not products.</li> <li>Yes. The deployment process should take care of deps.</li> <li>No, bundling dependencies is an anti-pattern. At worst case make our own local cache or proxy.</li> <li>When dependencies die, we can use a local cache. Or: dependencies never die.</li> </ul> <h2 id="docker">Docker</h2> <p><a href="https://docker.com">Docker</a> seals these questions shut — everything is snapshotted into an image, and you deploy that. This provides a layer of abstraction on top of the dependencies idea — Snapshot all the things.</p> <p>But still, for what’s written here, we’re talking pre-production-docker era here (which is, only a year and a half ago).</p> <h2 id="go">Go</h2> <p>Even without Docker, Go packs a binary which is self-contained. And the answers to the above questions, are:</p> <ul> <li>Go builds its dependencies into the binary, making a self-inclusive deliverable</li> <li>Deployment framework caring about dependencies doesn’t matter anymore</li> <li>Bundling dependencies doesn’t matter anymore</li> <li>Dying dependencies doesn’t matter, dependencies live within your source tree</li> </ul> <p>And even <em>with</em> Docker, we have no drama. A <a href="https://github.com/gliderlabs/docker-alpine">5mb image</a> plus your binary size, makes pulling an new image and starting up (and failing, when needed) crazy fast.</p> <h2 id="the-surprise-factor">The Surprise Factor</h2> <p>Go makes portable Code. Java made that possible too. However, Go makes for a different kind of portability. It doesn’t compile to every platform under the sun (yet), but it does build for x86 and ARM.</p> <p>Building for ARM means building for mobile, and <a href="https://www.raspberrypi.org">Raspberry Pis</a>.</p> <p>My <a href="http://amzn.to/1iJ9XQ3">tipping point</a> for using Go was when I looked into Python and C, for building my several ideas for home projects. I had to look at Python because it looks like that’s what the entire RPi community used, and I had to look at C because I found that a typical Python app took 27Mb of RAM blank.</p> <p>Obviously for the first Raspberry Pi model I had, I didn’t have a lot of memory. So, I decided to try Go, and I cleared up a day to do that, because I guessed cross compilation and ARM were going to be a nightmare and I really wanted to use Go (better yet, I didn’t want to use C as bad).</p> <p>The first 5 minutes passed and I cross-compiled and built a hello world Go binary, SCP’d it to the Pi, and it printed a ‘Hello world’ and exited. This was Go 1.0 or something of the sort.</p> <p>Not making peace with how smooth everything went — I spent the following 10 minutes making sure, and double-making-sure, that I copied the correct binary, and it truly was my own Go program that was running.</p> <p>I had a day to spare because everything worked perfectly, so I started working on what eventually became <a href="https://github.com/jondot/groundcontrol">Ground Control</a>.</p> <h2 id="go-is-about-forgiveness">Go is About Forgiveness</h2> <p>Let me tell you a story about forgiveness, and Go code.</p> <p>Go is verbose. It lacks generics, it adopts code generation as an escape hatch for many things the core language lacks.</p> <blockquote> <p>To everyone with experience — code generation is a bad smell, and this is a problem; and it should be.</p> </blockquote> <p>However, coming from Node.js code bases, with the dreaded callback hell and a particularly low quality factor for community packages (early Node days) a Go code base looks like heaven. So we forgive.</p> <p>Just when you are starting to get used to punching out very verbose Go code, you start noticing these verbosity issues you have overlooked; they bug you on a daily basis, and they are everywhere. But then, this kind of codebase would usually indicate you’re a bit more serious with Go. My guess is that you’d be at the same stage where you want to start doing concurrency work.</p> <blockquote> <p>You discover Go’s channels, its concurrency model, and its nonblocking I/O.</p> </blockquote> <p>Once again, you learn to appreciate it and become forgiving. By this time, the systems you’re developing are complex (in a good way), and so they <em>want</em> to be generic.</p> <blockquote> <p>You want to start building infrastructure for yourself.</p> </blockquote> <p>Generics, and lack of language abstractions start to hit you, hard. And once again, at the same time, you notice that your production environment is quiet. So quiet that it allows you to even ponder these things.</p> <p>You’re noticing everything you build is very robust, performant, without a special effort on your part. Additionally, you remember that the last commits and fixes you made were relatively easy because everything was super spelled out.</p> <blockquote> <p>You accept Go</p> </blockquote> <p>At this point, you accept Go.</p> <p>Code generation, its CSP concurrency model, its simplicity, its hyper-focused single-purpose tools such as go-vet, go-fmt, and the likes and make peace with the fact that by using Go, you’re building and getting accustomed to a colorful, vivid, tool set.</p> <p>You become forgiving, because strangely, you doubted Go at every crossroad, and it didn’t let you down.</p> Using Travis CI to Validate Awesome Lists Thu, 31 Dec 2015 00:00:00 +0000 Unknown https://rng0.io/blog/using-travis-ci-to-validate-awesome-lists/ https://rng0.io/blog/using-travis-ci-to-validate-awesome-lists/ <p>I maintain several awesome-style lists, namely: <a href="https://github.com/jondot/awesome-react-native">awesome-react-native</a>, <a href="https://github.com/jondot/awesome-devenv">awesome-devenv</a>, <a href="https://github.com/jondot/awesome-colorschemes">awesome-colorschemes</a>, <a href="https://github.com/jondot/awesome-weekly">awesome-weekly</a>, and <a href="https://github.com/jondot/awesome-beginners">awesome-beginners</a>.</p> <p>Some more complete than others, and being an OCD curator, some really serve my own needs.</p> <p>What I (and probably others) didn’t realize, is that since the web is constantly changing, these lists are subject to breaking — directly proportional to the number of links they host.</p> <h2 id="travis-ci">Travis CI</h2> <p>I’ve been using Travis CI for nearly every serious project I host on Github, and it is really valuable, especially when I have an open-source project which is really nothing more than an idea, execution, and good intentions of the original author and community.</p> <p>For my awesome-lists, to make sure they don’t break without my noticing, and to make sure each pull request contains a valid link, I thought of using <a href="http://travis-ci.org">Travis CI</a> along with a custom script to parse and verify links from a markdown document.</p> <p>Surely this is not really testing code, and I hope it still is blessed by Travis CI (thanks for this awesome free service for open-source projects, guys!) but still it can be looked at as a sort of an integration test.</p> <p>The validation script is pretty naive, at the expense of being dead simple and yet effective:</p> <pre data-lang="ruby" class="language-ruby z-code"><code class="language-ruby" data-lang="ruby"><span class="z-source z-ruby"><span class="z-meta z-constant z-ruby"><span class="z-entity z-name z-constant z-ruby">BASE_URI</span></span> <span class="z-keyword z-operator z-assignment z-ruby">=</span> <span class="z-meta z-environment-variable z-ruby"><span class="z-variable z-other z-constant z-ruby">ENV</span>[<span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-single z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&#39;</span>BASE_URI<span class="z-punctuation z-definition z-string z-end z-ruby">&#39;</span></span></span>]</span> <span class="z-keyword z-operator z-logical z-ruby">||</span> <span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-single z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&#39;</span>https://github.com/jondot/awesome-react-native<span class="z-punctuation z-definition z-string z-end z-ruby">&#39;</span></span></span> </span><span class="z-source z-ruby"> </span><span class="z-source z-ruby">doc <span class="z-keyword z-operator z-assignment z-ruby">=</span> <span class="z-support z-class z-ruby">Nokogiri</span><span class="z-punctuation z-accessor z-double-colon z-ruby">::</span>HTML<span class="z-punctuation z-definition z-group z-begin z-ruby">(</span><span class="z-support z-class z-ruby">Kramdown</span><span class="z-punctuation z-accessor z-double-colon z-ruby">::</span><span class="z-support z-class z-ruby">Document</span><span class="z-punctuation z-accessor z-dot z-ruby">.</span><span class="z-keyword z-other z-special-method z-ruby">new</span><span class="z-punctuation z-definition z-group z-begin z-ruby">(</span><span class="z-support z-function z-builtin z-ruby">open</span><span class="z-punctuation z-definition z-group z-begin z-ruby">(</span><span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-single z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&#39;</span>README.md<span class="z-punctuation z-definition z-string z-end z-ruby">&#39;</span></span></span><span class="z-punctuation z-definition z-group z-end z-ruby">)</span><span class="z-punctuation z-accessor z-dot z-ruby">.</span>read<span class="z-punctuation z-definition z-group z-end z-ruby">)</span><span class="z-punctuation z-accessor z-dot z-ruby">.</span>to_html<span class="z-punctuation z-definition z-group z-end z-ruby">)</span> </span><span class="z-source z-ruby">links <span class="z-keyword z-operator z-assignment z-ruby">=</span> doc<span class="z-punctuation z-accessor z-dot z-ruby">.</span>css<span class="z-punctuation z-definition z-group z-begin z-ruby">(</span><span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-single z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&#39;</span>a<span class="z-punctuation z-definition z-string z-end z-ruby">&#39;</span></span></span><span class="z-punctuation z-definition z-group z-end z-ruby">)</span><span class="z-punctuation z-accessor z-dot z-ruby">.</span><span class="z-support z-function z-builtin z-ruby">to_a</span> </span><span class="z-source z-ruby"><span class="z-support z-function z-builtin z-ruby">puts</span> <span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-double z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&quot;</span>Validating </span><span class="z-meta z-interpolation z-ruby"><span class="z-punctuation z-section z-interpolation z-begin z-ruby">#{</span></span><span class="z-meta z-interpolation z-ruby"><span class="z-source z-ruby z-embedded z-ruby">links<span class="z-punctuation z-accessor z-dot z-ruby">.</span>count</span><span class="z-punctuation z-section z-interpolation z-end z-ruby">}</span></span><span class="z-string z-quoted z-double z-ruby"> links...<span class="z-punctuation z-definition z-string z-end z-ruby">&quot;</span></span></span> </span><span class="z-source z-ruby"> </span><span class="z-source z-ruby">invalids <span class="z-keyword z-operator z-assignment z-ruby">=</span> <span class="z-punctuation z-section z-array z-ruby">[</span><span class="z-punctuation z-section z-array z-ruby">]</span> </span><span class="z-source z-ruby"><span class="z-support z-class z-ruby">Parallel</span><span class="z-punctuation z-accessor z-dot z-ruby">.</span>each<span class="z-punctuation z-definition z-group z-begin z-ruby">(</span>links<span class="z-punctuation z-separator z-sequence z-ruby">,</span> <span class="z-constant z-other z-symbol z-ruby"><span class="z-punctuation z-definition z-constant z-ruby">:</span>in_threads</span> <span class="z-punctuation z-separator z-key-value z-ruby">=&gt;</span> <span class="z-constant z-numeric z-integer z-decimal z-ruby">4</span><span class="z-punctuation z-definition z-group z-end z-ruby">)</span> <span class="z-keyword z-control z-start-block z-ruby">do</span> <span class="z-meta z-block z-parameters z-ruby"><span class="z-punctuation z-definition z-parameters z-begin z-ruby">|</span></span><span class="z-meta z-block z-parameters z-ruby"><span class="z-variable z-parameter z-ruby">link</span><span class="z-meta z-block z-parameters z-ruby"><span class="z-punctuation z-definition z-parameters z-end z-ruby">|</span></span></span> </span><span class="z-source z-ruby"> <span class="z-keyword z-control z-ruby">begin</span> </span><span class="z-source z-ruby"> uri <span class="z-keyword z-operator z-assignment z-ruby">=</span> <span class="z-support z-class z-ruby">URI</span><span class="z-punctuation z-accessor z-dot z-ruby">.</span>join<span class="z-punctuation z-definition z-group z-begin z-ruby">(</span><span class="z-variable z-other z-constant z-ruby">BASE_URI</span><span class="z-punctuation z-separator z-sequence z-ruby">,</span> link<span class="z-punctuation z-accessor z-dot z-ruby">.</span><span class="z-keyword z-other z-special-method z-ruby">attr</span><span class="z-punctuation z-definition z-group z-begin z-ruby">(</span><span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-single z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&#39;</span>href<span class="z-punctuation z-definition z-string z-end z-ruby">&#39;</span></span></span><span class="z-punctuation z-definition z-group z-end z-ruby">)</span><span class="z-punctuation z-definition z-group z-end z-ruby">)</span> </span><span class="z-source z-ruby"> <span class="z-support z-function z-builtin z-ruby">open</span><span class="z-punctuation z-definition z-group z-begin z-ruby">(</span>uri<span class="z-punctuation z-definition z-group z-end z-ruby">)</span> </span><span class="z-source z-ruby"> <span class="z-support z-function z-builtin z-ruby">putc</span><span class="z-punctuation z-definition z-group z-begin z-ruby">(</span><span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-single z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&#39;</span>.<span class="z-punctuation z-definition z-string z-end z-ruby">&#39;</span></span></span><span class="z-punctuation z-definition z-group z-end z-ruby">)</span> </span><span class="z-source z-ruby"> <span class="z-keyword z-control z-ruby">rescue</span> </span><span class="z-source z-ruby"> <span class="z-support z-function z-builtin z-ruby">putc</span><span class="z-punctuation z-definition z-group z-begin z-ruby">(</span><span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-single z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&#39;</span>F<span class="z-punctuation z-definition z-string z-end z-ruby">&#39;</span></span></span><span class="z-punctuation z-definition z-group z-end z-ruby">)</span> </span><span class="z-source z-ruby"> invalids <span class="z-keyword z-operator z-assignment z-augmented z-ruby">&lt;&lt;</span> link </span><span class="z-source z-ruby"> <span class="z-keyword z-control z-ruby">end</span> </span><span class="z-source z-ruby"><span class="z-keyword z-control z-ruby">end</span> </span><span class="z-source z-ruby"> </span><span class="z-source z-ruby"><span class="z-keyword z-control z-ruby">unless</span> invalids<span class="z-punctuation z-accessor z-dot z-ruby">.</span>empty? </span><span class="z-source z-ruby"> <span class="z-support z-function z-builtin z-ruby">puts</span> <span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-double z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&quot;</span><span class="z-constant z-character z-escape z-ruby">\n</span><span class="z-constant z-character z-escape z-ruby">\n</span>Failed links:<span class="z-punctuation z-definition z-string z-end z-ruby">&quot;</span></span></span> </span><span class="z-source z-ruby"> invalids<span class="z-punctuation z-accessor z-dot z-ruby">.</span>each <span class="z-keyword z-control z-start-block z-ruby">do</span> <span class="z-meta z-block z-parameters z-ruby"><span class="z-punctuation z-definition z-parameters z-begin z-ruby">|</span></span><span class="z-meta z-block z-parameters z-ruby"><span class="z-variable z-parameter z-ruby">link</span><span class="z-meta z-block z-parameters z-ruby"><span class="z-punctuation z-definition z-parameters z-end z-ruby">|</span></span></span> </span><span class="z-source z-ruby"> <span class="z-support z-function z-builtin z-ruby">puts</span> <span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-double z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&quot;</span>- </span><span class="z-meta z-interpolation z-ruby"><span class="z-punctuation z-section z-interpolation z-begin z-ruby">#{</span></span><span class="z-meta z-interpolation z-ruby"><span class="z-source z-ruby z-embedded z-ruby">link<span class="z-punctuation z-accessor z-dot z-ruby">.</span>text</span><span class="z-punctuation z-section z-interpolation z-end z-ruby">}</span></span><span class="z-string z-quoted z-double z-ruby"><span class="z-punctuation z-definition z-string z-end z-ruby">&quot;</span></span></span> </span><span class="z-source z-ruby"> <span class="z-keyword z-control z-ruby">end</span> </span><span class="z-source z-ruby"> <span class="z-support z-function z-builtin z-ruby">puts</span> <span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-double z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&quot;</span>Done with errors.<span class="z-punctuation z-definition z-string z-end z-ruby">&quot;</span></span></span> </span><span class="z-source z-ruby"> <span class="z-support z-function z-builtin z-ruby">exit</span><span class="z-punctuation z-definition z-group z-begin z-ruby">(</span><span class="z-constant z-numeric z-integer z-decimal z-ruby">1</span><span class="z-punctuation z-definition z-group z-end z-ruby">)</span> </span><span class="z-source z-ruby"><span class="z-keyword z-control z-ruby">end</span> </span><span class="z-source z-ruby"> </span><span class="z-source z-ruby"><span class="z-support z-function z-builtin z-ruby">puts</span> <span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-double z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&quot;</span><span class="z-constant z-character z-escape z-ruby">\n</span>Done.<span class="z-punctuation z-definition z-string z-end z-ruby">&quot;</span></span></span> </span></code></pre> <p>And the Travis setup should be something like this:</p> <pre data-lang="yaml" class="language-yaml z-code"><code class="language-yaml" data-lang="yaml"><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">language</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">ruby</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">rvm</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> </span><span class="z-source z-yaml"><span class="z-punctuation z-definition z-block z-sequence z-item z-yaml">-</span> <span class="z-constant z-numeric z-float z-decimal z-yaml">2<span class="z-punctuation z-separator z-decimal z-yaml">.</span>2</span> </span><span class="z-source z-yaml"><span class="z-string z-unquoted z-plain z-out z-yaml"><span class="z-entity z-name z-tag z-yaml">script</span></span><span class="z-punctuation z-separator z-key-value z-mapping z-yaml">:</span> <span class="z-string z-unquoted z-plain z-out z-yaml">bundle exec ruby validate.rb</span> </span></code></pre> <p>And then, on each commit a build runs. A failing build looks like this:</p> <pre data-lang="text" class="language-text z-code"><code class="language-text" data-lang="text"><span class="z-text z-plain">Validating 254 links... </span><span class="z-text z-plain">.........................................................F.................................................................................................................................................................................................... </span><span class="z-text z-plain">Failed links: </span><span class="z-text z-plain">- react-native-context-menu </span><span class="z-text z-plain">Done with errors. </span></code></pre> <p>And is nicely exposed with a Travis badge on my READMEs.</p> <p>Check this out live on my <a href="https://github.com/jondot/awesome-react-native">awesome-react-native</a> list.</p> A modern iOS App Icon and Image Workflow with Blade Wed, 16 Dec 2015 00:00:00 +0000 Unknown https://rng0.io/blog/modern-ios-app-icon-and-image-workflow-with-blade/ https://rng0.io/blog/modern-ios-app-icon-and-image-workflow-with-blade/ <p>There are a few steps in a typical Designer-Programmer exchange. This starts when either you (the developer) — are part of the team and the designer had finished design work, or when you (the developer) hire a designer or make contract work, and now you need to integrate the deliverables into your app.</p> <p>The following is an explanation on how I chose to attack this problem, and an open source tool called <a href="https://github.com/jondot/blade">Blade</a> I’ve built to make the solution work.</p> <h2 id="current-workflow">Current Workflow</h2> <p>You typically need an app icon, and image assets you would like to include in various resolutions, where a given image would be at 1x, 2x and 3x resolutions, and an app icon would be <a href="https://developer.apple.com/library/ios/documentation/UserExperience/Conceptual/MobileHIG/IconMatrix.html">a bit more involved</a>.</p> <p><img src="https://rng0.io/blog/modern-ios-app-icon-and-image-workflow-with-blade/1.png" alt="" /></p> <p>Here are the steps:</p> <ol> <li>(Designer) Design</li> <li>(Designer) Produce — cut sizes, adapt resolutions</li> <li>(Designer) Handoff — provide assets, and specs</li> <li>(Developer) Integrate — place each provided asset in its catalog, and resolution. Repeat until it looks good.</li> <li>(Developer) Identify mismatch — sum all glitches up, provide feedback to designer.</li> <li>Go back to either (1), (2) or (3) based on step (5)</li> </ol> <p>Here are all the things that can go wrong, and how Blade compensates for them:</p> <ol> <li> <p>(Designer) Producing sizes — can produce wrong sizes for wrong assets. Can use the wrong tool for the job. Can save in wrong formats, and so on. Errors can also be introduced by tooling even when the designer may cut manually (using Sketch for example), or automatically (using a Photoshop script or a dedicated “App Icon” <a href="http://makeappicon.com/">generator website</a>). <strong>With Blade</strong>: the designer produces and delivers just <strong>one size</strong> of the asset.</p> </li> <li> <p>(Designer/Developer) Handoff — human miscommunication. The primary cause in a lot of <a href="https://en.wikipedia.org/wiki/Mars_Climate_Orbiter">software fiascos</a>. although not as costly as a spaceship crashing, miscommunication between designers and developers still rots their workflow slowly and surely.</p> <p><strong>With Blade:</strong> reduce communication, by providing a single asset, of known size — the largest possible.</p> </li> <li> <p>(Developer) Integration error — placing the wrong assets in the wrong resolutions, fitting assets incorrectly, and so on.</p> <p><strong>With Blade:</strong> automates asset catalog creation, and takes that concern away from the developer.</p> </li> <li> <p>(Developer/Designer) Communication error fed back into the feedback loop. Injecting errors into a feedback loop quickly creates an <a href="https://en.wikipedia.org/wiki/Positive_feedback">avalanche effect</a>.</p> <p><strong>With Blade:</strong> even if there is a feedback loop, Blade enables skipping all of the production steps, because it runs them automatically. So basically it is — get a new asset, rerun Blade.</p> </li> </ol> <h2 id="blade-powered-workflow">Blade Powered Workflow</h2> <p>With Blade the previous error prone workflow is reduced to a safer one:</p> <p><img src="https://rng0.io/blog/modern-ios-app-icon-and-image-workflow-with-blade/2.png" alt="" /></p> <p>Between the lines, this new workflow also establishes these:</p> <ol> <li>1 size to rule them all. 80⁄20 best case.</li> <li>Automated steps 2–4</li> <li>A zero-friction process to repeat for providing feedback. Laziness as a natural optimizer.</li> </ol> <h2 id="blade-quickstart">Blade Quickstart</h2> <p>Using Homebrew:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> brew tap jondot/tap</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> brew install blade</span> </span></code></pre> <p>Now, within your project do:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> blade<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>init</span></span> </span></code></pre> <p>And take a look at the Bladefile that was generated for you.</p> <p>This is how working with Blade looks like (movie might take a while to load):</p> <p><img src="https://rng0.io/blog/modern-ios-app-icon-and-image-workflow-with-blade/3.gif" alt="" /></p> <p>If this is interesting for you, see the Blade <a href="https://github.com/jondot/blade">README.md</a> file, for the full power of Blade.</p> <h2 id="why-blade-works">Why Blade Works</h2> <h2 id="contents-json">Contents.json</h2> <p>Blade was created in the wake of the Apple Watch release. At the time no tool supported the newer Apple Watch form factors, and it only emphasized the need to be coupled to Xcode itself, and not an external website/tool. Using a generator website or a Photoshop script — you are at the mercies of these, waiting for their update in the best case, and in the worst case — making all the additions by hand.</p> <p>Apple saves each image catalog in their own format, in a file called Contents.json. Blade reads and generates this same format, and practically behaves like Xcode itself. This promises a slick integration and future-proof tooling — when a new iOS is released with new form factors, Blade doesn’t need to update, nor do you need to, just rerun to generate additional sizes.</p> <h2 id="trade-offs">Trade-offs</h2> <p>Every great solution and great design has strong trade-offs. In fact, when you can’t identify the trade-off a product or service make, that’s where <a href="https://en.wikipedia.org/wiki/No_Silver_Bullet">you should be cautious</a>.</p> <p>Blade makes the following trade-offs:</p> <ul> <li>The deliverable is a single big image per catalog. Every other size is a derivative of that. This means, you cannot provide a completely custom design for a exceptionally small/big variant of the asset. We also recognize that in this scenario, no other tooling will help you and in any case you should resort to a manual workflow.</li> <li>Blade works with a single file format — PNG. While first versions of Blade included SVG, we found that <a href="https://en.wikipedia.org/wiki/Lanczos_resampling">Lanczos</a>, as a resampling algorithm for scale reduction worked perfectly. We removed SVG support to avoid confusing users — SVG renderers out there make a poor job of rendering a pixel-perfect image, and including a headless browser would make rendering a lot slower.</li> <li>Blade works with Xcode, and for Apple technologies only. We chose not to support Android by default, however it should be easy to add it.</li> <li>Blade is meant for the developer. There is no UI, and you are encouraged to find you own convenient way of maintaining assets — storage and versioning.</li> </ul> <h2 id="summary">Summary</h2> <p>Blade is now powering all of my iOS app. Working with designers (one of which happens to be my Wife) have become a lot more productive, and in the case of my Wife — makes a happier marriage :)</p> <p>Feel free to submit pull requests or feedback on <a href="https://github.com/jondot/blade">the Blade repo</a>.</p> Technological Inflation Wed, 09 Dec 2015 00:00:00 +0000 Unknown https://rng0.io/blog/technological-inflation/ https://rng0.io/blog/technological-inflation/ <p>A <a href="https://twitter.com/kensodev/status/661246946946846720">discussion on Twitter</a> stirred up my neurons, and I felt this is a good time to put in writing things I’ve been discussing with people in corridors for a long while.</p> <h2 id="say-hello-to-technological-inflation">Say hello to technological inflation</h2> <p>Whether we like it or not, it is in my opinion we are in an age of <em>technological inflation</em>.</p> <h2 id="we-can-t-recruit-easily">We can’t recruit easily</h2> <p>Having a fresh memory of trying to recruit just 4 years ago and that of today, the experience is so different that it’s really hard to grasp. It is much harder to recruit today, because we can’t make good from bad.</p> <p>The amount of shallow experience and skillset out there is astonishing, and worse — it is acceptable because companies are rushing out to try and blitz-recruit just to strengthen their lines or get better valuation.</p> <p>In big companies, HR presses hard to get that painfully needed signoff, because , if you were to be a mid-level manager, how do you explain to high level management that 100% of people you interview fail?</p> <p>In small companies and startups, new recruits are being swapped</p> <p>out in an unpleasant frequency (to both sides) after those companies realize</p> <p>they didn’t do a good job clearing out the minefield of hype</p> <p>and buzzwords and _awesomeness_ from those candidates’ CVs.</p> <h2 id="we-can-t-make-our-case-properly">We can’t make our case properly</h2> <p>As engineers and architects, we feel that every choice should be reasoned about and proven to be the right one; or the smart one. And for a lot of cases, if not all, this line of thought is justified. As the saying goes, “we are the choices we make” — I believe this is true for every product we build, and specifically for the sum of technical choices we make.</p> <p>Due to that, we must understand that the <a href="https://en.wikipedia.org/wiki/The_Paradox_of_Choice">Paradox of choice</a> overwhelms engineers as well.</p> <p>For example, within the Ruby community (or any such <em>stable</em> community, and defining what is a <em>stable</em> community is a subject for another post entirely) you pretty much are safe with most of the choices you make.</p> <p>However, in a typical front-end community, you are so completely lost; and after you made your (by all means correct) choice, chances are</p> <p>it will be the wrong one after six months. If you are just picking up a front-end stack, do you start out with Angular? <em>but</em> Angular 2.0 is killing 1.x…, how about React? would that be a smarter move? <em>but</em> Angular has plenty of talent out there already…; and that never ends.</p> <p>The most painful part is, that in our micro cosmos — <em>hype fuels the paradox of choice,</em> And that is destructive. It becomes a vicious feedback cycle. Not only you don’t know what to choose, you also may end up choosing something that is half-baked, bad quality, or unreliable in the long run.</p> <p>And then it gets worse. Even when you know what’s right, you are peer pressured into that moment’s hype; and everyone keeps parroting the same agenda. Sounds familiar?</p> <h2 id="behavioral-economics-applied-to-tech">Behavioral economics applied to tech</h2> <p>I suspect even the tech community exhibits some of the behavioral economics patterns like any other community.</p> <p>Here’s one to think about: Open source is ‘free’, and ‘free’ makes <a href="http://www.amazon.com/gp/product/0061353248/ref=as_li_tl?ie=UTF8&amp;camp=1789&amp;creative=390957&amp;creativeASIN=0061353248&amp;linkCode=as2&amp;tag=test0554-20&amp;linkId=SRU4OPOB7HUX3Q3E">people do irrational things</a> I love open source, and I am personally fueled by doing open source (for reasons mentioned below). But let’s ask this for a moment:</p> <ul> <li>Do we make the same judgement when adopting an open-source product as we do a product which we paid a license for?</li> <li>Do we judge it under the same constraints?</li> </ul> <p>The answer is — probably not. But now that we know how we behave, we can counter it (seriously, that book is great). And we can judge open-source products with the same pair of glasses as any other product, with a healthy engineering approach (more on that in the end).</p> <p>Funny thing is, that I wouldn’t even think about that back in 98' when I built my first open source project; as a person who comes from a modest home</p> <p>with limited resources, my motivation was that I wanted to be able to help out others like me.</p> <p>But 17 years of doing open source later, open source becomes almost an industry, it’s a whole different beast where the concepts of altruism and freedom are no longer the sole substance that structure it.</p> <p>As any rich ecosystem it is exploding with options, some of them are completely awesome, and some of them are not.</p> <p>Sadly, the practice of decision making and making rational choices, didn’t get that kind of love. In such a vibrant environment, sometimes,</p> <p>we are overwhelmed, pressured, and pushed into a direction which doesn’t match our intuition.</p> <h2 id="there-is-hope">There is hope</h2> <p>Technological inflation, is where our sense of value and the act of judging value becomes useless. <em>Everything</em> is awesome.</p> <p>Luckily this is how engineers resolve this:</p> <ul> <li>Measure twice cut once. Every tool, and every product, should prove itself to your problem domain.</li> <li>Time proven over _just released_. All products need time to mature; no matter what they say. If people are counting on your product, give a serious thought before adopting edge.</li> <li>Some things are just the same but in different packaging. Learn to identify those and stick to patterns and techniques, not tools and frameworks.</li> <li><a href="https://www.youtube.com/watch?v=iwUR0kOVNs8">Test all the f__king time</a> (or really, make a best effort). Prefer platforms that let you do that more easily. Once you have that in place, switching always costs less.</li> <li>Listen to your gut, <a href="http://joshkaufman.net/explore-exploit/">and test your gut every so often</a>. If you have solid and good experience with a product or a tool, that must amount to something.</li> </ul> <h2 id="the-end">The end</h2> <p>In some point in time, you get to a stage where you start <a href="http://bitquabit.com/post/the-more-things-change/">seeing history repeating itself</a>.</p> <p>You see it everywhere. You just don’t bother mentioning it anymore or writing about it. You conserve energy, smile in acknowledgement, point people gently at the right direction, and keep on going. And naturally these are the voices that don’t get heard as much, sans the occasional blogpost (you’re reading it).</p> <p>If you smiled and acknowledged this, welcome to being old. And wise. And sometimes grumpy. Embrace it :)</p> <p>Thanks <a href="http://twitter.com/kensodev" title="Twitter profile for @kensodev">@kensodev</a> for the inspiration and <a href="http://twitter.com/ketacode" title="Twitter profile for @ketacode">@ketacode</a> for making the <a href="http://berzniz.com/post/132422546311/to-follow-or-not-to-follow-the-hype-that-is-the">first post</a> about it, you guys rock!</p>