Backup Options for Open Source SQL Databases

2021-08-10T00:00:00+00:00

I've been trying to setup a kind of mini datawarehouse for my homelab, and I'm spoiled for choice. As I'm wont to do on this blog, I'm going to do a technical survey of the various options that seem interesting to me.

Criteria</h2>

Convenient Setup - installing and configuring and maintaining that config over time is just too hard. Let's use an off-the-shelf (base) docker image from a reputable source. We don't want to have lots of docker containers for this either. Ideally, just one container, not like 5 docker-compose services.</li>
General Purpose - we might be doing time series analysis, full text search, point queries, or crazy joining. I don't want to lock in right now nor use many databases. I want one single database to setup that can handle whatever kind of read load I throw at it at very low scale with okay performance</li>
Massive Ecosystem - I'd like to plugin this database into Superset, various custom glue code applications, and make the value of the data I collect as great as possible. Big ecosystems generally have lots of stackoverflow, good official docs, etc.</li>
Convenient high-frequency backup and restore - For a upcycled homelab situation, servers get unplugged, spontaneously restart, or just fail all the time. I want to leverage cloud storage for backup/restore to minimize data loss at rest. When I plug the server back in, I want to make sure I can easily recover as much data from before the power incident as possible.</li> </ul>
So, while I keep telling myself I love NoSQL databases (and I think that's still true?), I'm really looking at the most well known databases here and their upstart/mostly compatible twins. Sorry Mongo and DynamoDB! As much I'd love to use Presto/Hive or even Clickhouse to be able to query directly against something like S3 or minio, all of those database options have far too many moving parts and would require 2-5 docker containers just for a hello world style beginning. Once we've windled down the list this far, then probably the ease & power of the backups really starts to look like the differentiating factor.
The Contenders</h2>
Based on the above criteria, I'll be looking at:

Postgres</li>
MySQL</li>
MariaDB</li>
CockroachDB</li> </ul>
Flavors of Backup</h2>
It's really hard to compare the various backup options offered by different databases without having a detailed understanding of the various mechanisms for backup that exist and the inherent tradeoffs in each. Before diving into the databases, I'm going to try to compare and contrast these different styles of backup.
Flavor</th> Trigger</th> Whole dataset?</th> Locking</th> FS Tweaks Needed</th></tr></thead>

Logical Backup</td> Normally cron</td> Optionally</td> Table</td> No</td></tr>
File System Snapshot</td> Sys admin</td> Mandatory</td> Server</td> Yes</td></tr>
Replication</td> DB Activity</td> Impractical</td> None</td> No</td></tr> </tbody></table>
Replication Sensation</h3>
What I'm calling "replication" here has many different flavors itself. For all flavors of replication, for all practical purposes, you must have one of the other forms of backup to anchor the changes since the last backup. Restoring then requires a sequence of commands of find the appropriate full backup, and then replay the replication data "on top of" the full backup.
Postgres leverages it's own Write-Ahead-Logs ("WAL") to enable consistent crash recovery, and it exposes the ability for server admins to automatically "archive" these WALs with arbitrary scripts. Those scripts could upload WALs to S3, etc. However, if uploading the WALs ever fails, it can cause postgres to become unavailable. If you give up on a WAL, then your backup will have lost data.
Replicating the logs to another server and forming a cluster is also a form of backup, in that it offers a "backup" server in case anything happens to the master. It can also be used to reduce extra load on a leader during a full backup, or even theoretically used to restore a new master. Since replication doesn't really allow us to ease do backup to a commodity cloud storage system, I'm going to rule out this kind of replication.
Mysql accomplishes it's replication to other servers using the binary log or "binlog" that contains all mutation-inducing queries. Like Postgres leveraging WALs for incremental backups, MySQL can be configured to leverage the binlog for incremental backups to a file. Unlike the WALs, MySQL is less vulnerable to network blips taking down the database server. However configuring and testing and monitoring the binlog.
While a PiT restore capability would be great to setup, this is an advanced features that won't work without some kind of full backup solution already in place. 
"Logical" Backups</h3>
Logical backups can dump everything in a database to a text file filled with SQL statements. At a later time or on a separate machine, those statements can be executed to restore the database to its state at the time of the backup. They are potentially large to store, but easy to restore and widely supported. Logical backups can be run remotely over a TCP or UDP connection in case we want to run an external cronjob or manually trigger this without SSH access. Conveniently, a full logical backup can run on selective tables of a given database.
In the case of MySQL, `mysqldump</code> can even output CSV or other formats. Postgres and MariaDB do similar things. Technically, things can change while you run the backup, so its not guarranteed to be perfectly consistently.`
`You'll still need to setup a cronjob to run mysqldump and friends, then upload to S3...which means installing an S3 client and configuring that as well. Here's a prepackaged scheduled backup to s3</a> for Postgres that looks promising.`
`CockroachDB actually has full support for backups to S3 on a schedule baked in</a>. The winner here is CockroachDB, since it's the simplest. Just run a command one time, and CockroachDB will make the backup on a schedule and upload it for you.`
Snapshot Backups</h3> Postgres, MySQL and MariaDB all provide facilities for "snapshot" backup of the underlying file system with various caveats. The main problems are (1) a database server in motion might be snapshot at a bad moment that is inconsistent and (2) a database server might some key information in RAM, but not yet on disk. To mitigate these issues, the client must lock the database and then tar</code> up the various files of interest while holding that lock in an ongoing connection. Finally, the client must release the connection. It might be possible to script that with bash, but it would annoying. I'm not sure why a tool that does this for you is not simply included in these database products. Using similar techniques, file system level backups can also be taken using ZFS or BTRFS (or LVM too). However, those snapshots can't be exported to cloud storage. I looked at docker checkpoint</code> as well, but this won't work because it only captures the running processes memory, not disk. Sadly docker commit</code> would probably also require downtime to backup the database, though in this case would be acceptable. However, the new docker image would still need to exported for cloud backup and would likely be even larger than a logical backup. Conclusion</h2> Based purely on the backup criteria, my small dataset, and some tolerance for data loss over complexity, I think I'm going to use the open source "core" CockroachDB's full backup schedule and just do full backups hourly with a 1-2 week retention. Fun with Histograms - Prometheus Basics 2021-07-27T00:00:00+00:00 Building on my previous learning exercises, I need to learn how to query Prometheus</a> so that I can work with production systems running on Kubernetes. For large scale datasets, I find histograms to be an excellent tool for summarizing and visualizing throughput and latency. However, this is all a bit confusing and new to me in Prometheus. Summaries, Histograms, Oh My!</h2> Prometheus has two metric types for histograms -- Summaries and Histograms. Summaries</h3> Summaries are calculated client side, meaning the CPU must dedicated cycles to computing these values it could be serving your customers. Exact percentiles are precomputed and stored in a ready-to-use state in Prometheus. However, you can't calculate any new percentiles that weren't explicitly calculated ahead of time. You can think of a Summary as being stored inside Prometheus something like this: You can see in the diagram that the metric name provided in the application code http_response_duration_ms</code> is stored with explicit labels for the exact percentile (called quartiles</code>). You can retrieve any of those time series with an instant selector, but you can't apply any new functions. Here's a sample PromQL for the median value from the diagram above: http_response_duration_ms{quartile="0.5"} </code></pre> Sadly, because the percentiles are all precomputed, you can't combine this value from multiple kubernetes pods at query time. Some other system would have to calculate a Summary ahead of time...so we're not going to talk about Summaries anymore. Histograms</h3> Histograms are calculated server side, within Prometheus itself at query time. Prometheus stores histograms internally in buckets that have a max size (labeled le</code>), but no minimum size. You must configure the number and max size of each bucket ahead of time. Each bucket time series will contain the count of observations that was less than or equal to its le</code> value for a given timestamp. Things look quite a bit different. Each bucket is stored in a separate metric suffixed with _bucket</code> and the maximum value for that time series is in the label le</code>. There is always a largest bucket with infinite maximum value {le="Inf"}</code> which will always have the same value as _count</code>. Because you can't meaningfully use the buckets directly AFAICT, you generally use the function histogram_quantile()</a> to estimate whatever quantile you want. The accuracy will be based mostly on the sizes of buckets you choose in the client. The more and smaller the buckets, the more accurate but the more data required to store and compute values. Querying Histograms</h2> Let's start playing with the histogram_quantile()</code> function. Here's a simple PromQL query against the histogram for the diagram: histogram_quantile(.99, rate(http_response_duration_ms[1m])) </code></pre> Even though this is the simplest usage of histogram_quantile()</code>, there's kind of a lot. Let's break it down: .99</code> - this is the "rank" or the "percentile" being called in a range from 0 to 1</li> rate()</code> - we'll talk more about this, but basically this converts a counter metric into a usable form</li> [1m]</code> - we're aggregating at 1m intervals here</li> </ul> Counters and rate()</code></h3> If you read the docs, you'll see that counters are "monotonically" increasing. That means that a given metric name with a given set of labels will always increase, but the problem is graphing a line that always goes up just isn't very helpful. So, rate</code> will break down time slices, see the increase during the time slice, and make that the value. Let's take some observations at different times of a counter as an example: Time</th> Counter Value</th></tr></thead> T1</td> 3</td></tr> T2</td> 5</td></tr> T3</td> 7</td></tr> T4</td> 7</td></tr> T5</td> 17</td></tr> </tbody></table> So, if we apply rate()</code> to this time series, then the values will be changed like this: Time</th> rate(counter)</code> Value</th></tr></thead> T1</td> 0</td></tr> T2</td> 2</td></tr> T3</td> 2</td></tr> T4</td> 0</td></tr> T5</td> 10</td></tr> </tbody></table> So, basically, because sending histograms to Prometheus generates a whole bunch of different counters (specifically the _bucket</code> metric names) we normally need to use rate()</code> first on the get usable data. Aggregating Histograms by Series</h3> So, let's say you have a lot of pods running a service and you add labels to all your metrics that includes the pod. For this particular metric, you've also included the http status code. However, you now want to build a dashboard out of one metric showing the 95th percentile of something, for example http_response_duration_ms</code>. If you naively use the code I shared above you'll see a time series graphed for the combination of each unique pod and status code!! Let's not do that. Instead try this: histogram_quantile(.99, sum(rate(http_response_duration_ms[1m])) by (le, status_code)) </code></pre> The main thing that's new and different here is that sum(..) by (..)</code> will look for any series that share the same labels {le=.., status_code=..}</code> and add all their values together into a single series. Let's take an example using a single sample/timestamp across many series to show what happens: le</th> pod</th> status_code</th> value</th></tr></thead> 500</td> pod1</td> 200</td> 5</td></tr> 500</td> pod2</td> 200</td> 5</td></tr> 1000</td> pod3</td> 200</td> 10</td></tr> 1000</td> pod4</td> 200</td> 10</td></tr> 1000</td> pod5</td> 404</td> 1</td></tr> </tbody></table> After running the promql above, the above series sum up any series that have the same le</code> and status_code</code>, so the end result looks like: le</th> status_code</th> value</th></tr></thead> 500</td> 200</td> 10 (5+5)</td></tr> 1000</td> 200</td> 20 (10+10)</td></tr> 1000</td> 404</td> 1</td></tr> </tbody></table> Now, you'll see one series per status code across your whole cluster. If we had any other labels (like endpoint</code>, service</code>, etc), those would also disappear after the aggregation above. Don't Forget Your le</code>!</h3> Behind the scenes, at query time, the histogram_quantile()</code> function will secretly look for all labels of {le=..}</code> on the provided series. Before I understood this, I kept hitting errors saying: No datapoints found. </blockquote> Once you know that histogram_quantile()</code> requires the time series passed in to have the labels {le=..}</code>, this actually makes sense. This is why anytime we combine histogram bucket series, we must be careful to preserve the le</code> labels. Conclusion</h2> Using histograms as a lens, we've dug deep into how Prometheus: expects data to come from client applications</li> stores time series</li> converts counters into usable time series using rate()</code></li> exposes histograms via histogram_quantile()</code> function</li> allows us to combine different labels for a metric name</li> </ul> Hopefully, this lets you start doing interesting analysis of your production systems! Java LSP for Neovim - Java Basics 2021-07-15T00:00:00+00:00 This time in my continuing forray into Java, I'm going to setup a Language Server Protocol for Neovim. Quick Notes: "LSP" stands for Language Server Protocol</a>, and its an awesome standard invented by Microsoft for VSCode designed to help everyone stop reinventing the wheel. The gist is that anyone figures out how to do things like goto-definition for a programming language, all editors that speak the protocol can immediately start using that one implementation. I use this acronym excessively in this article, so I figure it's worth explaining here first :)</li> Probably not important, but just FYI I'm doing all this on Linux. Generally things will actually be easier for you on Mac or Windows (sorry BSDs) than for me.</li> </ul> TL;DR - OpenJDK11 + coc.nvim + coc-java</h2> After trying some other things, I eventually found that Conquer-of-Complete (aka "CoC")</a> really does JustWork (tm). Although, it doesn't advertise the Java LSP implementation, it's out there</a>. The nice thing about this setup is that "coc-java" installs the underlying LSP (a small piece of Eclipse) for you and configures everything. The other thing I like about CoC compared to the nvim-lspconfig project is that CoC allows me to configure key bindings once for all LSPs. Enough blathering, let's do this! 1. Install OpenJDK</h3> Since I'm on Ubuntu 18.04 ("Bionic"), the most recent JDK available was 11, but really anything greater than Java 8 would have sufficed: $ sudo apt-get install -y openjdk-11-jdk </code></pre> The coc-java plugin will need to know where this is. I chose to tell the plugin like so: set -xU JDK_HOME /usr/lib/jvm/java-1.11.0-openjdk-amd64 </code></pre> Yes, that's fish shell, not bash. It's the equivalent of adding export JDK_HOME=...</code> in my .bashrc and then sourcing immediately. 2. Install coc.nvim plugin</h3> Inside my ~/.config/nvim/init.vim, I added this to install the plugin using vim-plug</a>: Plug 'neoclide/coc.nvim', {'branch': 'release'} </code></pre> Then restart Nvim and run :PlugInstall</code> 3. Setup Keybindings</h3> The most interesting bits of functionality to me to set key bindings for in the init.vim I've captured below: nmap <silent> gd <Plug>(coc-definition) " GoTo definition nmap <silent> gr <Plug>(coc-references) " GoTo references nmap <leader>rn <Plug>(coc-rename) " Rename symbol nnoremap <silent><nowait> <space>a :<C-u>CocList diagnostics<cr> " Show all errors/warnings nnoremap <silent><nowait> <space>c :<C-u>CocList commands<cr> " Kitchen sink of all LSP commands nnoremap <silent><nowait> <space><S-s> :<C-u>CocList -I symbols<cr> " Fuzzy search for ALL symbols </code></pre>4. Building Muscle Memory</h3> Being a vi user is all about building habits so that we can achieve various actions reflexively. For me that means picking a small number of keybindings and practicing them intentionally. One new thing I'm trying is to write down the above keybindings on a piece of paper taped to my monitor. My hope is that this will help me memorize these shortcuts a bit more "organically" and I can eventually throw the paper away or replace with new bindings I want to start memorizing. Every 6-12 months, I'll weed out keybindings that didn't stick. Here's hoping that most of these keybindings are here for the long haul! Simple Logging Façade for Java (slf4j) - Java Basics 2021-07-12T00:00:00+00:00 I've recently started on a new project where I need to write nontrivial amounts of Java, which is great! Except that I've never written any Java before, outside some data structure classes. So I'm going to try help myself retain and deepen my knowledge of various Java topics (beyond just syntax & semantics) by keeping short notes of a few paragraphs here on my blog to help me feel like I'm making progress. Simple Logging Façade for Java (slf4j)</h2> Basically, the idea is to decouple the logging implementation from the log statements in your application until deploy time. At deploy time, you just add a Jar with the logging implementation to your $CLASSPATH</code>, and the logging works. Or, if the Ops team wants to use a different logging framework, then they just swap in a new Jar in production and you the developer don't have to change anything. It just works. That's cool! Rust has standardized on a similar logging framework (probably inspired by slf4j) which delegates the logging implementation to compile time. However, switching loggin implementations in Rust would require recompiling the entire application and would require at least one line of code change in the Cargo.toml. Win for slf4j on the deployment side! Adding slf4j to your Java Code</h2> There's a few different patterns here, the log format string (like Python): // Java logger.info("Something went wrong: {}", badThing); </code></pre># Python logger.info("Something went wrong: {}", bad_thing); </code></pre> These format strings are kinda the same as Rust too: // Rust info!("Something went wrong {}", bad_displayable_thing); </code></pre> Another style is the fluent logging that basically uses the "builder" pattern to progressively construct a log message piece by piece: logger.atDebug().addArgument(myPojo).addArgument(dbConnection).addArgument("Hi Mom!").log("Args: 1={} 2={} 3={}"); </code></pre> You can pass virtually anything to addArgument()</code> and it will be coerced to a printable representation. You must use the at<LEVEL>()</code> methods of Logger</code> to use the fluent API, and .log()</code> to flush it. Copy-pasting from the docs</a>, you can also label an argument with a key instead of only a position. By default, these two log statements will output the same string: // using classical API logger.debug("oldT={} newT={} Temperature changed.", newT, oldT); // using fluent API logger.atDebug().addKeyValue("oldT", oldT).addKeyValue("newT", newT).log("Temperature changed."); </code></pre>Creating the slf4j logger instance</h3> Since we don't know during development what the logging implementation will be, we need to make this a runtime decision by the slf4j framework. That means using a factory. Let's see it in action: import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class HelloWorld { public static void main(String[] args) { Logger logger = LoggerFactory.getLogger(HelloWorld.class); logger.info("Hello World"); } } </code></pre> Notes: Logger</code> - this is the parent class that all logging implementation classes will subclass. It's the actual logging object you call .log()</code> on</li> LoggerFactory</code> - this is how slf4j figures out at runtime which implementation to give you</li> </ul> It's often handy to just have an instance of Logger</code> ready to use, so in at least one codebase at work, I've seen instances of setting up the logger as a static member of the class, something like this: import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class HelloWorld { private static final Logger logger = LoggerFactory.getLogger(HelloWorld.class); public static void main(String[] args) { logger.info("Hello World"); } } </code></pre> Now, we don't have to pass loggers around or make sure each method creates a logger. Instead, every method on this class will now have access to a logger at any time. Farewell to Mail-In-A-Box 2021-05-01T00:00:00+00:00 I've been using Mail-In-A-Box for quite some time now, and while it's served me well for quite a while, I feel like it's time to get more involved in managing my personal infra. Gratitude for Mail-in-a-box</h2> Joshua Tauberer</a> has done an incredible job of making a self-hosted email server affordable and accessible, updating the project to incoporate the latest protocols and generally inspiring confidence that upgrades will JustWork(tm). He's been incredibly generous with his time, and I doubt I would have gone down this path without the high quality of work he's painstakingly lavished on this project for so many years. In many ways, he represents the finest traditions of open source. Things I Dislike about Mail-in-a-box</h2> TLS problems Maybe it's just me, but I can't seem to resolve this symlink loop issue</a> that takes down my server either every 24hrs or every 3 months, depending. Newer versions of Mail-in-a-box use MTA-STS</a> which means that I must fix TLS issues immediately or I can't even read already delivered email messages. Once you combine that with rate limits at LetsEncrypt</a>, you have an amazing recipe for chronic downtime. Limited tests and an ambivalent attitude towards continuous integration</a> means I can't contribute back proof that my problem is fixed while keeping the existing functionality working. Also, it just means maintainers spend a ton of energy manually testing everything. Email usability problems There are 2 big problems here: (1) addresses are only auto-suggested during email authoring if they are manually added to the address book, and (2) the calendar integration isn't google calendar friendly. Specifically, if someone sends me an invite to an event, and I click "yes", the event won't appear on my calendar. This makes it hard to coordinate with others. Backup and Monitoring The builtin monitoring tool, Munin</a> can't be configured to send alerts without forking the project or hacking the code. Personally, I've not worked with Munin much in my day job and would rather use something like telegraf + influxcloud, or grafana cloud, etc. Additionally, it could just be me, but I've never been able to get duplicity working and so I effectively have no backups. Unconfigurable The project is very upfront about this: Please note that the goal of this project is to provide a simple, turn-key solution. There are basically no configuration options and you can’t tweak the machine’s configuration files after installation. </blockquote> This is a great way to structure a project to maximize the chances someone can install and use this software, even as a novice to the world of email infrastructure (like myself). Its very beginner friendly and it minimizes differences between users so that the commmunity can help each other as much as possible. But for me as a professional software engineer, it's limiting to have a rich ecosystem of plugins for Roundcube or other applications just out of reach. Monolithic Architecture Because Mail-in-a-box tries to completely takeover and run self-contained and unconfigurable on a single VM, everything about it becomes a spof</a>. Now, your DNS server is also your monitoring endpoint, your LDAP, your webmail client, and your email server. You can't use tools like Kubernetes to do bin packing and run other things on the box. You also can't leverage your experience with terraform or ansible to configure things. Things I love about Mail-in-a-box</h2> The builtin status checks are incredible! The forums and community is really active. All various applications fit nicely together. Upgrading Mail-in-a-Box generally means pulling in new features, bugfixes, and security enhancements without much effort on my part. A new kind of Mail-in-a-box</h2> So, what if we wanted to make a personal email server that modular instead monolithic? What if we created a series of containers that could be deployed together in a known-good configuration, or extracted individually and reused in other projects? What if such a project had behavioral tests and a modern CI pipeline? That would be a project I'd be proud to put my name on. Proposed Architecture</h2> Since we're looking at supporting IMAP and SMTP protocols, we'll need a layer-4 load-balancer/reverse proxy/gateway thingie. SMTP and IMAP protocols have a kind of banter going on, where clients and servers send one or more lines to each other over a persistent TCP connection. If bots could chat, this would basically be that kind of protocol. C: <open connection> S: * OK IMAP4rev1 Service Ready C: a001 login mrc secret S: a001 OK LOGIN completed C: a002 select inbox S: * 18 EXISTS S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft) S: * 2 RECENT S: * OK [UNSEEN 17] Message 17 is the first unseen message S: * OK [UIDVALIDITY 3857529045] UIDs valid S: a002 OK [READ-WRITE] SELECT completed C: a003 logout S: * BYE IMAP4rev1 server terminating connection S: a003 OK LOGOUT completed </code></pre> Since managing LetsEncrypt via Mail-In-A-Box has a been weak spot for me, ideally we'd use ZeroSSL (another free ACME server with higher rate limits). Even better, let's do TLS termination at the load balancer and separate encryption concerns from IMAP/SMTP stuff. To keep things modular and friendly to clustering solutions, we'll want to manage sets of stateless docker containers separately for SMTP and IMAP bits and bobs, have a service discovery solution that integrates with the load balancer, let the load balancer manage the TLS certs, and some kind of cluster manager schedule the SMTP and IMAP workloads. We also need a stateful store of email data (mbox? mail queue? things like that I think) and some way to specific users and their credentials. Finally, we want to make sure we can build and test the clusters ability to send/receive email in a CI pipeline, and monitoring hooks to integrate with standard tools. Now we're cooking! Next Steps</h2> My plan is to write up my steps towards making this cluster a reality! Next stop, get postfix working. Review of Diagrams-as-Code for mdBook 2020-07-03T00:00:00+00:00 When maintaining documentation for a software project, you need to keep diagrams to describe everything. However, those diagrams will go out of date fast if they aren't easy to maintain. Rather than keep the raw diagram originals in a separate file, embedding diagrams inline using some other diagramming language (like dotviz) can make it easy to change the words and diagrams together and ease the maintenance burden. These days there are a lot of tools in this space. This post is an attempt to try out and compare these different tools strengths and weaknesses. Criteria for Tool Selection</h2> For this post, I'm specifically interested in tools that can integrate with the mdBook static site generators and enable the author to embed the diagram code inline with the markdown text. There are lots of other tools that allow authoring a diagram in code separately from your markdown, but I'm not interested in those for this post. I've fallen in love with mdBook</a> for general purpose documentation. There's basically nothing to configure and you get: a static site generator</li> automatic, offline, fast search</li> automatic reloading while writing & editors</li> statically compiled binary</li> </ul> Additionally, mdBook supports plugins which can add in support for all kinds of goodies, not least diagrams as code inline in your markdown. The Contenders: Diagram as Code Tools</h2> Here's the top tools I'm considering: mermaidjs</a> - v0.4.2</li> plantuml</a> - v0.5.0</li> svgbob</a> - v0.1.1</li> graphviz</a> - v0.0.2</li> </ul> Installation</h2> preprocessor</th> binary size</th> compile time</th> precompiled</th> additional requirements</th></tr></thead> mdbook-plantuml</td> 5.8MB</td> 8m45s</td> windows</td> plantuml server/executable</td></tr> mdbook-mermaid</td> 4.3MB</td> 8m03s</td> yes</td> none</td></tr> mdbook-svgbob</td> 3.3MMB</td> 7m44s</td> no</td> none</td></tr> mdbook-graphviz</td> 2.5MB</td> 6m55s</td> no</td> graphviz</td></tr> </tbody></table> These compile times are very unscientific. Some crates were downloaded from the network and others were cached locally. Don't read too much into that, except that if you're going to build your mdbook in a CI pipeline, you'll want to leverage the cache you don't need to recompile these dependencies on every run. The winner here is mdbook-mermaid because one can forgo the compile time and simply download a pre-compiled binary from the Github releases page for a wide variety of platforms and because it doesn't require any additional install steps. Mdbook-svgbob comes in second place over-all for compiling its image rendered inside itself, and not needing any other dependencies. The PlantUML preprocessor is interesting because it can be configured to connect to a local or remote plantuml server. The advantage here would be if your team already has access to a plantuml server, you wouldn't need to set up the plantuml server yourself. Improvements for Compile-Time & File Size</h3> All four crates depend on clap</code>, serde</code>, pull-down-cmark</code>, pull-down-cmark-to-cmark</code> and mdbook</code>, mostly because this is the recommended approach</a> in the mdbook user guide chapter on preprocessors: By pulling in mdbook as a library, preprocessors can have access to the existing infrastructure for dealing with books... Then each chapter of the Book can be mutated in-place via Book::for_each_mut(), and then written to stdout with the serde_json crate... The pulldown-cmark crate implements a production-quality event-based Markdown parser, with the pulldown-cmark-to-cmark allowing you to translate events back into markdown text. </blockquote> While these dependencies are high quality, there are some smaller crates that would suffice for this use case: miniserde</a> instead of serde + serde_json</li> pico-args</a> instead of clap</li> </ul> If mdbook itself exposed a small "core" crate that simply exported the Preprocessor</code> trait and other types, that might reduce the file-size/compilation time of all of these crates as well. How Images Are Produced</h2> preprocessor</th> format</th> when</th> separate files crated</th></tr></thead> mdbook-mermaid</td> svg</td> browser javascript</td> no</td></tr> mdbook-svgbob</td> svg</td> mdbook build</td> no</td></tr> mdbook-plantuml</td> svg or png</td> mdbook build</td> yes</td></tr> mdbook-graphviz</td> svg</td> mdbook build</td> yes</td></tr> </tbody></table> From a setup and publishing point-of-view, mdbook-mermaid is probably in first place here. The rendered is simply copying your MermaidJS diagram from the code fence into a <pre class="mermaid">{}</pre></code> html tag, and defering the SVG rendering until page load time. So, theoretically, this processor should have the smallest impact on mdbook build time. (Of course, this would make your readers page-load time worse, but that assumes anyone's reading your pages anyway!) Having separate SVG files created is a slight win here. You can send someone a URL to this image, whereas if the image only exists inline with the HTML, its slighltly more difficult to get access to. Breadth of Diagram Styles</h2> preprocessor</th> styling</th> flow chart</th> graphs</th> sequence diagram</th> class diagram</th> cloud</th> SQL schema</th> shapes</th></tr></thead> mdbook-svgbob</td> yes</td> yes</td> yes</td> yes</td> probably</td> yes</td> probably</td> yes</td></tr> mdbook-mermaid</td> yes</td> yes</td> some</td> yes</td> yes</td> yes</td> yes</td> no</td></tr> mdbook-plantuml</td> yes</td> yes</td> ?</td> ?</td> yes</td> yes</td> yes</td> ?</td></tr> mdbook-graphviz</td> ?</td> yes</td> no</td> ?</td> yes</td> probably</td> yes</td> no?</td></tr> </tbody></table> Honestly, this is just hard to compare. svgbob tries to infer outlines of shapes from ascii art, so it can create just about anything, but it requires the writer to be willing to play with random characters as an art medium, instead of as a rigidly defined illustration protocol. The big upshot of this is that even when the svgbob illustrations are viewed as plain text, they should convey the same impression. The downside is diagrams produced this way are likely much more time consuming to update and maintain. MermaidJS seems to just steal whole swathes of syntax and semantics from graphviz, but offers just a subset of that functionality. Then it adds many other forms of visualization that I don't believe are part of graphviz (like pie charts). PlantUML is a newish dialect of UML designed to be used in documentation. It supports many different forms of visualization. Despite the code not looking much like the final image, graphviz, plantuml, and mermaid make it relatively easy to refactor your image by tweaking the code behind it compared to svgbob. If suddenly you need to add (or remove) a box from the middle of your diagram, or decide to start using squares everywhere, you'll have retype nearly the whole svgbob diagram -- but probably not for the others. The mdbook-graphviz only supports the "dot" visualizations, so puts the plantuml and mermaid preprocessors ahead in this category. Prettyness Factor</h2> I have to say, the svgbob diagrams seem to be the most beautiful, and PlantUML to be the ugliest. Mermaid comes in second place to me based only my personal taste. YMMV. svgbob diagrams MermaidJS Sequence Diagram GraphViz linux diagram PlantUML sequence diagram Ecosystem</h2> Interestingly enough, PlantUML is implemented on top of GraphViz, which was released in 1991 by AT&T Labs. So, there's plenty of examples, tutorials, and chatrooms where you can find support for both of these tools. Despite the rather meager search trends for MermaidJS compared to the top two, there have been two main contributors over the life of the project (since 2014) and there is a Gitter chatroom. Many developers have contributed tens of commits to the project over its life. Just in the last week, 16 pull requests were merged. Also, mermaidjs has an awesome live code editor</a> you can play with. Highly recommend! By comparison, svgbob really has only 1 developer (with about 10 others who've contributed a few commits a piece) and no activity in the last month. It might be tough to get support when you encounter rough edges here. Despite these worrying signs, there is a nice code editor</a> here as well. Conclusion</h2> Personally, I'll be using MermaidJS with mdbook going forward, but I'll definitely play with svgbob as well! Returning Trait Objects 2020-01-22T00:00:00+00:00 This is a fairly basic Rust syntax issue that I've run into several times. Based on unknowable runtime conditions, I will return one of several different return types from the same function. Also, this return type must use methods from two traits. How can I express this in Rust? Solution</h2> fn foo(runtime_condition: bool) -> Box<dyn BoilerTrait> { if runtime_condition { Box::new(Type1) } else { Box::new(Type2) } } </code></pre> BoilerTrait</code> is a trait that requires the traits I actually care about (ex: UsefulTrait</code> and Debug</code>).</li> Type1</code> and Type2</code> are concrete types that both implement trait BoilerTrait</code></li> I can use methods from any of the traits required by BoilerTrait</code> on return value of foo()</code> transparently</li> </ul> Failure 1: return impl Trait</h2> I mistakenly thought impl Trait in return position would be the way to handle this case. Here's code that does NOT compile: fn foo(runtime_condition: bool) -> impl BoilerTrait { if runtime_condition { Box::new(Type1) } else { Box::new(Type2) } } </code></pre> The compiler complains: error[E0308]: if and else have incompatible types ... = note: expected type `std::boxed::Box<Type1>` found type `std::boxed::Box<Type2>` </code></pre> For the return impl Trait syntax, the compiler will only allow us to return one concrete type. Why Can There Be Only One?</h3> I think there are two advantages if I understand this right: Static dispatch and runtime performance improvements</li> More flexible public API contract for consumers of a library</li> </ol> It's a middleground for library authors to avoid committing to a specific type in their public API without sacrificing the performance improvements of static dispatch. I think you might even be able to swap out the return type at compile time using cfg!()</code> macros or what-not. Cool! But, that's not my use case. Failure 2: Multi-Trait Object</h2> More code that doesn't quite compile: fn foo(runtime_condition: bool) -> Box<dyn UsefulTrait+Debug> { if runtime_condition { Box::new(Type1) } else { Box::new(Type2) } } </code></pre> The compiler tells us: error[E0225]: only auto traits can be used as additional traits in a trait object --> src/main.rs:12:33 </code></pre> I have no idea why you can't name multiple normal traits in a trait object. That's why we have to make a super trait BoilerTrait</code> that requires the traits you actually want. Use Type Annotations for Temporaries</h2> When trying something more complicated than the examples above, temporary values weren't automatically recognized as the trait object I had in mind...It seems like the type inference could be better at this case, but in rust 1.40.0 you need explicit type annotations for this case. First, the code that does NOT compile fn foo(runtime_condition: bool) -> Box<dyn BoilerTrait> { let temporary = if runtime_condition { Box::new(Type1) } else { Box::new(Type2) }; temporary } </code></pre> The compiler complains that the conditional returns different types. This is confusing because, the exact same expressions were returned just fine a moment ago without the let binding. In our previous examples, without the temporary, the final expression from the conditional is returned from the function, which is explicitly annotated as -> Box<dyn BoilerTrait></code> so the compiler doesn't have to do any guessing about the type. However, it does have to guess about the type of temporary</code> because there's no type annotation when the value is bound. Once the compiler decided that temporary</code> is going to be a concrete type, we've lost the trait object. The solution is easy, just add type annotation to the let binding: let temporary: Box<dyn BoilerTrait> = ...; </code></pre>Full Solution</h2> Here's a fuller working code example and a playground</a>: // useful trait 1 use std::fmt::Debug; // useful trait 2 trait UsefulTrait { fn useful(&self) {} } trait BoilerTrait:UsefulTrait + Debug {} #[derive(Debug)] struct Type1; impl UsefulTrait for Type1 {} impl BoilerTrait for Type1 {} #[derive(Debug)] struct Type2; impl UsefulTrait for Type2 {} impl BoilerTrait for Type2 {} fn foo(runtime_condition: bool) -> Box<dyn BoilerTrait> { let temporary: Box<dyn BoilerTrait> = if runtime_condition { Box::new(Type1) } else { Box::new(Type2) }; temporary } fn main() { // Call method from trait UsefulTrait foo(true).useful(); // Call method from Debug trait dbg!(foo(false)); dbg!(foo(true)); } </code></pre> Tracking Personal Finances through Email 2019-11-22T00:00:00+00:00 Mint</a> redefined personal finances for me: automatic categorization, syncing with my bank, real time alerts, mobile and web apps. In the aftermath of the Equifax breach, I felt scared about having all my financial history in one place with millions of other people. It was time to take control of my financial data. A Little Background on Me</h2> My income was beginning to feel stretched thin as my family grew from 2 to 4. Childcare costs climbed to 50% of our income, and we couldn't save money. I feel strongly that Facebook, Google, and Amazon are undermining civil society, democracy and capitalism, so I doubted there was huge earning potential for me somewhere else. Oh, and, we wanted to remodel our house...Suddenly, I really needed a budget. Spoiler alert: I still need a budget. Lots of blog posts on these topics explain magic solutions, but this post is an odyssey and confessional through all of my failed approaches to date. It's 50% therapy, 30% competitve landscape, and maybe 20% technical problems and solutions. Requirements</h2> Our ultimate goal was to formulate and follow a realistic budget to find enough money for our big expenses. Requirements: Download and classify transactions automatically</li> Help with spending: Reactively -- alert about overspending</li> Proactively -- check remaining budget before buying</li> </ul> </li> Banks, credit cards, and retirement accounts</li> iPhone, Android, MacOS</li> No 3rd parties</li> Empower my non-tech life partner</li> </ul> After reading the NY Times article about technology-based domestic partner abuse</a>, I've started taking that last bullet point more seriously. Each of those stories boiled down to one partner setting up the smart home, and then post-breakup using aspects of the smart home to harrass the other person. So, if both partners are familiar and comfortable with all the important tech in their life, this abuse should mostly be mitigated. For personal finances, this means there must be a simple enough user interface accessible on both our smart phones. Nice to haves: Double-entry accounting</li> Export / import to ledger</a></li> Good visualization</li> Customizable categorization -- an area where I wanted more than Mint provided</li> </ul> Looking at Alternatives</h2> There's loads of options out there. I only looked at open source options that let me own the data. That eliminated things like LunchMoney, YooNeedABudget, and other paid options. ledger / hledger / etc</h3> I spent quite a bit of time tracking my finances with these tools, and they are incredible. I felt so on top of where my money was going when I used these. I'm convinced that double-entry accounting in ledger-style is what I want personally under-the-hood for personal finances. You can teach yourself a lot of accounting just from the documentation in these projects, and I've that helpful in my professional life and in various roles in my HOA and other civic groups that handle money. Ledger & Friends Pros:</h4> Double-entry accounting</li> No 3rd parties</li> Portable data format</li> Mature technology</li> Excellent documentation</li> Active community</li> Can support any currency or commodity</li> </ul> Ledger & Friends Cons:</h4> Can't pull data from my bank / credit cards -- manual step here</li> No real time alerts</li> Single device / terminal or desktop website UIs</li> Limited categorization functionality</li> Can't imagine my partner using this</li> </ul> GnuCash</h3> If you prefer GIMP</a> to Photoshop, you might prefer GnuCash</a> to Quicken. I haven't used this tool extensively, but it seems like it has a community behind it, and it's the only tool I looked at that can help with tax preparation. Folks running a small business are probably more the target market for this tool, rather than personal finance folks. GnuCash Pros</h4> Private - No Third Parties</li> Double Entry Accounting</li> Active development and community</li> Strong support for Quicken exports</li> Feature rich</li> Desktop GUI (and Android too)</li> Multi-currency</li> Internationalization</li> </ul> GnuCash Cons</h4> Basically the same as ledger & friends... Firefly III & Build-a-Mint-Clone & Friends</h3> There's a ton of tools like Firefly III</a>. Some of them are double entry accounting, others are not. Many of them can import CSVs and Quickbook files. Some of these tools are desktop apps (not multi-device) and others have web interfaces. Fava</a> is the closest to what I want in this category. It can be accessed via a website (so that counts as multi-device) and it's a near cousin to ledger. None of these work for me because none of these tools can sync data from banks without using a 3rd party API. Also, I'm not sure I could get my partner to use any of them. Summary of Alternatives</h3> No alternatives had these three features at the same time: Automatic download of transactions</li> No third party data sharing</li> Multi-device</li> Compelling for my partner</li> </ul> I felt like I had an opportunity here to solve my own problem and possibly bring a new product to the market. Dead End 1: React & RethinkDB</h2> Back in 2016, I started using RethinkDB</a> to store the transactions, and React to render charts and graphs about my transactions. I figured out how to import CSVs from one of my bank, and started calculating net income. This project was a lot of fun, but didn't really solve the hardest problem at all: downloading financials automatically. Around this time, RethinkDB Inc shutdown, and so I removed the backend and made this an entirely JS + HTML project without a server. I realized you could probably build a 100% offline client to handle this, if you were willing to make the user send you CSV files from the bank themselves. My work and home life got very difficult, and I gave up for at least a year. Learnings</h3> Client side JavaScript could probably do a lot of stuff I hadn't consider before</li> Babel/webpack/frontend toolchains are the absolute worst for prototyping if you don't regularly work on the frontend full time (hello vue.js</a>!)</li> Downloading transactions should be the first problem I solve because everything else is probably easy</li> </ul> Dead End 2: Curl</h2> There were awful fires in California in 2017</a>, and then again in 2018</a> and our windows didn't seal at all. I worried about exposing our children to Beijing-level air-quality</a> year after year, and so we wanted to replace our windows. The costs were staggering to me, and given our childcare costs, there was no way we could afford it without serious cost-cutting. I decided to try making a budgeting tool again. Remembering that downloading transactions was the hardest thing, I start thinking about this first. I used the network tab in my browser to see how to login to the bank and download a CSV of transactions. Then I tried copying those commands as curl calls. I figured that the cookies would have all the details necessary to prove I was logged in and that the bank's server software would see curl commands copied from my browser as equivalent to the real thing. I was completely wrong. After many nights and weekends, I discovered the credit union used an ancient ASP server framework that serialized and hashed all of the session state, including initial session ids, inside various form elements on the page. Every time I clicked on a link or button inside my browser on their website, the client side code would take the form state from the previous request scattered across form elements, and then produce a new hashed response code in a mangled field name that varied, and return that to the server. Without this complex dance of tightly coupled client and server code, it was impossible to move through the UI. I could only find this information after a trip down the wayback machine when trying to identify the server framework identifying some cryptic fields being based to backend. So, the chances of me being able to find working documentation for the exact version of this ancient proprietary framework used by my local credit union were slim. Learnings</h3> My bank's website might be slightly harder to hack than I first thought -- go bank!</li> I would need to use a real browser to login and download transactions</li> </ul> Dead End 3: Selenium</h2> I spent the next two or three weeks of 15min sessions writing up some Selenium</a> scripts in Python to login to the bank and download transactions. Having solved this and run the code at least twice, I felt sure I had nailed this. Port to Rust</h3> I love Rust</a> a lot and wanted to spend my spare time writing software in Rust. So I started porting the Python to Rust. Unfortunately, Rust libraries for Selenium</a> aren't nearly as feature rich and mature (or ergonomic) as those in Python</a>. The maintainers I chatted with were super helpful and willing to accept PRs and feedback, and I was able to get something sort of working. Challenges with Selenium</h3> Having now run these scripts over a tethered connection 5-10 times in a row per session (there were many sessions) and over wifi in various locations, I started hitting random failures. Sometimes, it was just timeouts for finding a link on the page were too low, but other times the bank would put another page in the login flow to announce a promotion (student loans or something) and my script would break 1 in 4 times. Sometimes at night or when tethered or running the script a lot, the bank would require an SMS second factor authentication. Score one for bank security noticing I was a bit odd for logging in over and over! State Machines</h3> After listening to a podcast about the Automat library</a>, I realized that I could solve these problems with a state machine. I basically need to make a graph of state transitions, and iteratively complete a state transition (enter my username and click "login") then detect the next state. (I'm not quite sure if this technically fits into the definition of a "state machine", but its close.) I realized I could solve the 2fa problem by connecting an IFTTT SMS-trigger</a> on my phone to a webhook to my webapp</a>, and then parse out the text message for the one time passcode. This could be incorporated into the state machine. It kinda worked! Sometimes, as long as I didn't do the SMS thing. But I had several accounts with other financial institutions I wanted to scrape. I worried about how I could possibly find the mental health and time to write another version of this for every single bank out there. Code Quality Issues</h3> My code was really hard to work with. I hacked it together when I was at my most tired, often forcing myself to code in the middle of the night or during my 20 minute bus ride to work. It was mostly a 1000 line function that panicked without real error messages when anything went wrong. Code was repeated all over the place and there were no tests. To solve these problems, I spent many hours architecting the program. I tried to follow a Domain Driven Design</a>, with various abstractions to represent Banks, Transactions, Categories and Budgets. There were RFC-style plans for everything I wanted to accomplish. But the more planning I did, the more dread and resentment I felt at trying to tackle such a large project without getting enough sleep or having anyone to help me, and trying to do so with a difficult programming language and small ecosystem. I was just super unhappy with life due to work and family and neighbor stuff going on, and I gave up for many months. Learnings</h3> I needed to be more surgical in both planning and writing code</li> YAGNI</a> -- I spent too much time planning out a fully baked program when I hadn't proven out the basic approach was scalable to multiple financial instutions</li> I've been at this for years already (no one on Hacker News seems to spend years doing anything), so I need to take time to relax and sleep and only do this when it's fun</li> As in UX testing, Selenium can be really fragile, especially if screenshot support is missing from your library</li> </ul> Current (Not-Yet-Working) Approach: Email</h2> At this point of burnout and frustration, I read several posts on Hacker News (which I can no longer find) about using transaction alert emails to track personal finances. Each of these blog posts focused on the simplicity of this approach (especially compared to Selenium), but no one wrote about the security implications. Instead of entrusting your bank password to questionable browser hijacking code running unsupervised inside your bank account, you configure your bank to send emails about every transaction. Basically every bank supports this, so you don't have to wait for banks to have proper APIs. Assuming you setup a separate email address for this purpose with a different password than your bank account, then there's nothing for a hacker to exploit. All they can do is read about your finances, not steal all your money! Also, Selenium can't fuck up your bank account on accident either. Once I thought through the security, I completely gave up on Selenium. IMAP -> MySQL -> Grafana</h3> Learning from my recent burnout on this project, I decided to write the minimum amount of code as fast as possible. The best advice for this I've found is to reuse as much open source software as possible. Basically, I would write a script in Rust to check emails then parse them for transactions details, then write that information to MySQL. I could visualize that information using Grafana</a>. No web server, no frontend code, very little math. I got this working in under a month of nights and weekends. But as soon as I tried to visualize the data, it was clear that the data was wrong. Really wrong. I was pretty discouraged. I felt like Grafana wasn't going to be a great option for building an easy interface for my partner to use. Given the data wasn't right, I decided to remove Grafana and focus on cleaning the data. Data Cleanup</h3> Digging into the data, I noticed that any transfers between accounts were being double counted. So, that's sort of a tricky problem. I still haven't really solved that one yet. Instead, I decided to focus on one single account, and tackle cross-account issues once I was more confident in the basic data. Looking into one account at a time, I started finding lots of problems: Lots of transactions were missing</li> There were extra transactions, with totally wrong amounts</li> All the dates were wrong</li> Some transactions were doubled</li> </ul> This past month is the point were I'm finally starting to find really interesting details about how banks send emails. So, this is sign I'm possibly doing something right? Missing Transactions</h4> My first pass at parsing emails involved very detailed regexes. These were often only right for some of the messages. Instead, I've taken to just writing regexes that use a lot of blah blah .* blah blah</code> in them. This seems to work much better. Also, multi-part emails kicked my ass for a long time. Like, there would be HTML and text versions of a message inside the same email. But also some messages from the same bank would straight up text or just HTML. So, I was finally able to unify the code to handle that by making a list of parts of an email (the list might only have one element), then iterating over the list and stripping any HTML tags out of HTML emails. Then I would take the first element from the list for parsing with my regexes. This seems to work great! Extra Totally Wrong Transactions</h4> Turns out these emails were actually messages about declined transactions. These email notifications would make for great real time notifications once I take the time to parse those emails and build them into the application somehow. But also it seems valid that declined transactions are not listed in my transaction history on the website. This is area of functionality I hadn't anticipated and that I probably couldn't build using Selenium. All the Dates were Wrong</h4> In banking, there's generally two dates associated with any transaction: the posted date and the effective date. The posted date is when the bank first learned about this transaction. The other date is called the "effective" date, and this is when the bank formally "clears" the transaction. If the bank doesn't clear the transaction immediately, it could be considered "pending" for up to 2 or 3 business days, and it probably didn't affect your balance until it clears. All that to say, for debit card transactions on checking accounts, I get emails for the posted date so that's why the dates in MySQL always a day or two before what my bank shows. This might be a WONTFIX issue, unless I can think up a clever way to solve it later. Doubled Transactions</h4> Related to the last section, sometimes I get a second email that's identical to the first email, when the transaction "clears." But not always. So, I think I've only just solved this problem in the last week. Before I insert a new row into MySQL, I construct an ID in my script. The ID is a hash of only some columns I'll insert: Transaction date</li> Description</li> Amount</li> Account</li> </ul> It doesn't include the email date nor email id (did you know emails have uids?? I only recently learned that from the IMAP protocol). Now, when there are both posted and effective date emails for the same transaction, I get MySQL errors about duplicate IDs. Success! Not Quite Success</h5> So It turns out, in rare cases there really are multiple transactions on the same day with the same description and amount and account. Gah! How do you distinguish between these cases? You can't. Not using transaction emails. Daily Account Balances</h3> However, banks also provide daily account balance emails, so we should at least be able to detect when the actual bank balance isn't the sum of the transactions from the previous day. Also, tracking the balance of an account (even if its only daily) is super valuable as an independent data point for my application. How Close Am I?</h2> It's really hard to say. I've been focused on creating a dead simple cli in Rust, and I'm learning interesting things about the IMAP protocol, accounting, and Rust. My data is quickly becoming more accurate, but it's still not right. I'm also learning how to work on side projects more effectively by breaking down work into multi-hour tasks, and how to prioritize the right next step. It's actually fun and I have some momentum. I hangout with friends and (rarely) rage program anymore. But nothing comes for free. I still mostly sacrifice sleep or time with friends or that just using ledger directly so I can work on this project. Honestly, I haven't 100% proven that emails can accurately track my net worth. Once that (hopefully) pans out, there's a killer interface for personal finances and setup an LLC and marketing and yada yada to do. But I'm still optimistic, and for now I'm going to keep chugging away because this is one way I choose to try to create meaning for myself (and there's still nothing out there that quite does what I want). Web Analytics without Google - A Just Tracking Theory 2019-07-29T00:00:00+00:00 I've been thinking about adding some kind of analytics to this website for quite some time, but I've been feeling very conflicted about it. Here's my current thoughts on why tracking might not be evil...at least sometimes! Just War</s> Tracking Theory</h2> Although philosophers and statespersons debate the morality of war, marketing people don't ever really consider this in my experience. For philosophers, there are two questions. One, is war ever justifiable? Two, under what conditions is a country justified in going to war? In an age of endless data breaches and tampered elections, I believe a moral theory underlying why it's ever morally permissible to do web analytics is necessary. Without such a theory and conditions under which tracking is morally permissible, content creators run the risk that society will demonize our efforts to share our knowledge and talents with the world. I have two basic arguments for why I believe tracking is moral under certain conditions. The Friendly-Store-Keeper Argument</h3> A vaguely human shaped bundle of black clothing waddles into your newstand, and puts a gloved hand on a copy of the local newspaper. After leafing through a few pages, the masked bundle sets down the paper and picks up the sports section of a national paper. Finally, the bundle grunts through a voice-hiding mouth-piece and buys a pack of gum at the counter with exact change. You smile politely, ask if they want a receipt, and finally wave farewell as the bundle waddles away. What's odd about my imaginary story is that we don't expect anywhere near that level of anonymity when conducting business in person offline. But, if there were no web tracking, this is a good analogy for how a website owner would see activity on her site. Just as its normal and not creepy for a shopper keeper to notice my skin color and age when I walk into her store, likewise its reasonable for individual websites to learn something about my visits to their site. For example, I assume a website owner will know what pages I visited, perhaps if I'm a first time visitor or a returning guest, etc. The Revenue-By-Other-Means Argument</h3> I really like the Economist. I pay a few hundred dollars a year to receive one issue per week chocked full of British humor and surprising stories from across the world. Last week's top article: Operatic Cross Dressing in China. However, if I didn't pay the Economist, when would the reporters find time to do the research for the paper, or take all the tiny little pictures, or reflect on the sad state of the world in 2019? Answer: they wouldn't. In fact, my subscription alone doesn't even fully compensate all the expenses of the paper, so the Economist (very tastefully) solicits advertising and receives revenue by another means. Similarly, there are many incredible resources online. If no one pays to use them, there won't be as many. Tracking provides information necessary to convince advertisers to subsidize something valuable to society that is hard to finance otherwise. Consumer Espionage</h3> Google Analytics is doing something different. To return to the friendly store keeper analogy, while I think Google should mind their store, I object to their following you down the block, into neighboring stores, and then into your house to stare at your photos. The scope of surveillance from Google is truly staggering -- the microphone in your pocket, the camera in your phone, every email and text message and conference call, your location at every moment and even the questions on your mind as you search for answers. Building a detailed profile of a person surreptitiously from many sources resembles nothing so much as espionage. While spies secretly collect information, they do it to manipulate, physically harm, or discredit their opponents. Google's goal of manipulating one's purchasing behavior seems much closer to a sort of "consumer" espionage. Why should you care? Because if you use Google Analytics on your blog or website, you've become a spy for Google by allowing them into your store. Be the Friendly Store Keeper You Want To See in the World</h2> How can an individual site owner ensure their tracking is more like a friendly store keeper, and less like consumer espionage? Just don't use Google Analytics, and you'll stop feeding the corporate surveillance machine. It turns out, at least for smaller blogs and websites, this isn't all that hard to avoid. There are a handful</a> of</a> alternatives</a> that you can even run yourself, but any data analytics platform which is not Google is a huge win for privacy. With Great Data, Comes Great Responsibility</h3> But, in an age of data breaches and tampered elections, it's not sufficient to avoid surveillance. We have to be responsible for the data we collect. I think we're just starting to think about this as a society, but to me personally it means at least these additional things: Keep your</s> other people's data safe</li> Don't collect more data you have to</li> </ul> I'm not a security expert, but I do think that most security issues stem from a lack of concern about security. If everybody cared just a little bit more to follow basic best practices from the likes of OWASP</a> and NIST</a>, everyone's data would be a little bit safer. In terms of minimizing data collection, analytics tools too often dictate this. However, I think its possible to change the status quo and demand more privacy from our analytics tools. Next Time</h2> This was the first part in a mini-series on tracking without Google. Next up, what metrics are even worth tracking and a review of alternatives to Google Analytics. Please Don't Deply Before You Test 2019-07-26T00:00:00+00:00 In the small to medium software teams that I've been on, our development cycle is inefficient and error prone at a conceptual level. Code changes are fully tested only after becoming the source of truth for the team, causing rollbacks that corrupt git history. Too often, developers are unable to determine whether a piece of code will actually work in production before they ship their code. I propose several practices to more fully test your code before deploying it to your customers. Setting the Stage: Typical Development Cycles</h2> Laptop -> CI -> Approval -> Staging -> Production 8hrs 30min 8hrs 1hr Years </code></pre> Length of time branch exists at each phase of development cycle In this kind of a workflow, the only production-like environment used to test software happens immediately before sending your code to your customers. What, Specifically, Is Bad about Staging Right Before Production?</h2> Problem 1: Prod is just different, man</h3> There are often significant differences which emerge only in a production-like infrastructure. I once spent days developing and testing delete functionality for DynamoDB, adding in conditions to the delete to ensure atomicity in cases of writes happening concurrently, only to discover that the mocking library for DynamoDB had a different version of conditions than the DynamoDB running in AWS. If I could have easily run this code against real DynamoDB, I would have discovered this issue mucher earlier. While this kind of issue doesn't happen to all developers everyday, it's not uncommon among my peers and it can double or triple the time required to complete even simple tasks. These kinds of problems aren't limited to proprietary cloud systems either. I've personally encountered foot guns when using open source components, such as Celery, Redis, Django's ORM, and MongoDB -- because these are the infrastructure facing technologies I've run lately. The production infrastructure wasn't properly mimicked anywhere, except maybe in staging. Problem 2: Dips on Staging</h3> Waiting for access to staging is another problem. At one of my jobs, we manually tested each branch after merging. I remember rolling out a big change to add SSL support for a large new customer. This change had the potential to break all images, stylesheets, javascript and basically everything if I didn't get it right. So, I took staging from the team for a few hours to test it manually, but blocked the team from deploying. Anyway, I finally got it right on the third try. For nearly every team I know of where there's a staging environment, I hear about production deploys being blocked due to the need to deploy various experimental changes before hitting production. Problem 3: Bugs Going Viral</h3> In teams of up to a hundred engineers working in the same codebase, if wrong assumptions about infrastructure reach production, its entirely possible for one developer to pull down the authoritative branch (let's call it "master") during the window of time that this wrong assumption has not yet been rolled back. Defective code like this can go viral and reoccur even after a rollback. Its actually even worse than that. If automated tests run against a staging environment only after merging to master, its entirely possible that an unwitting developer pulled in the bad code which never reached production. At some point later, she discovers that her branch doesn't pass tests for code unrelated to her changes. At one company I recently worked for, this problem became so widespread that the QA team built tools to toggle individual tests via a Google spreadsheet so that they could increase developer velocity. A few times, I'd spend 30 minutes to an hour diagnosing a bug unrelated to my own code, open up a PR to fix the unrelated code, and discover that another PR or two had already been opened and reviewed (but perhaps not yet merged to master) before mine was opened. Fixing Software Development</h2> For nearly all these issues I've identified above, some kind of test already existed but the test ran on staging too late. What if every change could land on stage automatically before merging to master? What if you could manually poke at this stage at your leisure, without any time pressure or blocking other engineers release? Fewer bugs and greater velocity. Solution 1. Make a "Staging" Per Branch</h3> Expense</th> Hourly</th> Annual</th></tr></thead> "Web Developer" in Kentucky</td> $36.29</td> $75,487</a></td></tr> m5d.24xlarge on-demand EC2 instance in us-west-1</td> $6.38</a></td> $13,023</td></tr> </tbody></table> The hourly wage is calculated by dividing the yearly salary into 52 weeks, and then dividing each week into 40 hours</li> The annual cost of an m5d.24xlarge is calculated for the a 51 week year of 40hrs per week</li> </ul> Even the cheapest labor is significantly more expensive (more than 5x) than the priciest cloud computing. But, if you hire experienced web developers in the Bay Area (or other coastal cities), the labor costs could be 2 to 3 times that of the our colleagues in Kentucky. And the EC2 instances we'd be testing on is likely much less expensive (do you actually need 384 GiB of RAM and 96 cores for your production workloads?). Any work an engineer is doing that can be replace by a cloud computing instance should be. So, if all of our developers are sitting idle, waiting for a chance to test their work on a production-like environment, that looks like an ideal point to arbitrage the compute / engineer cost difference. If all of our engineers can spend less time waiting for tests to complete, and more time delivering features or fixing bugs, that means the company could be getting more value out of the workforce they already have without a significant outlay. Objection! Our production infrastructure is hard to replicate</h4> Congratulations! You just found an argument that gives you budget to do some real devops on your production infrastructure. Bonus: you'll get really good at disaster recovery if every branch is mini-rollout of your production infrastructure. The details of how you do this are not something I can unpack in this blog post right now, but there are lots of other great resources on this starting with the 12 factor app</a>. Objection! How could an expensive EC2 instance possibly make any difference?</h4> If you're not already in the cloud, it's the same principle: you'd want to buy a set of hardware which is identical to your production infrastructure and configured using the same set of automation for your on-premises servers. Oh, you don't automation for your on-premises hardware? See previous paragraph. Either way, the closer your development environment (and particularly your CI environment) is to production, the more effective the tests you already have will be. By deploying and running ALL tests against a production like environment, you will not bogged down with hardware or instance-type specific quirks. As we discussed earlier, if staging is the only production-like environment, and all tests need to pass through staging, you've got a single point of failure and a potentially limiting factor in your CI/CD pipeline. To solve both problems, eliminate the notion of a permanente environment called "staging" and create ephemeral environments for each branch that's developed. Objection! This still seems really expensive</h4> There's a lot you can do to cut costs on this operation. Here's a few ideas: Invest in autoscaling so that when your development environments are idle, they scale to 0</li> When the branch is deleted, automation should be able to clean up that ephemeral branch to reduce costs</li> If the whole company works in a few adjacent timezones, turn off all the machines for the night via automation. Bonus: employees will avoid burnout by going home at a "reasonable" hour</li> </ul> These techniques, alone or in combination, should keep your costs minimal while providing strong guarrantees about the code you ship to production. Objection! What about databases, credentials to vendor integrations, etc?</h4> This part depends heavily on your context, so I can't provide the right answers for your case. However there are strategies you can try. For relational databases, you can spin up a database cluster that's an exact mirror of production, but only used by development environments. Each of your branches can create a separately named set of databases for it and your automation can run migrations or pull a recent snapshot from production (probably best to sterilize any sensitive data in there first!). For DynamoDB or other similar proprietary NoSQL databases, you can just create a duplicate AWS/GCP/Azure account, and namespace your database. For other third party integrations, I'd recommend making a separate account and reusing credentials where possible. Speak to your vendor to find out how other customers are solving this problem -- if the vendor is smart, they've find a way to solve your issues. None of these techniques are new or particularly clever. They just take an investment of time. Solution 2. No More Sneaky 3-ways</h3> When your Github repository allows merge commits (a "three-way merge" technically), this means that its very possible code from your branch won't work after its merged into master. There's very little guarranteed about the logic or syntax of your code after a 3 way merge. However, if its merged to master, your bugs are going to go viral -- even when the tests are already written that would catch these bugs! Rebasing Is the Answer!</h4> However, if your branch must be rebased against master and is only allowed to be a fast-forward merge, then you have guarranteed that the code in your PR will be exactly the same code in master. This means the test result will be the same for anything that relies purely on the code. This is huge! The key point here is not how to use git correctly, but how to guarrantee that a good starting point for other developers to base their own work upon. Rebasing Isn't Really The Answer</h4> In fact, at a certain scale, simply doing rebases probably won't work anymore. The problem will become that developers can't rebase and finish tests as fast as code is being deployed to production and merged to master. This leads to a vicious cycle of rebasing endlessly against the current HEAD</code> of master, like racing up an escalator going down. OpenStack addresses this problem by testing combinations of branches together all the time to find combinations that will work before they are merged to master. They've custom built a tool called zuul</a> for this. From chatting with friends at Facebook, I think they do something like queue a series of changes, and put these changes into a deterministic order. By ordering the change requests, they've effectively gotten the same guarrantees as a rebase-based workflow. Netflix seems to tackle this problem by breaking things into microservices so that there aren't huge codebases collaborated on by thousands of engineers together. But, Seriously, Rebasing Kinda Is the Answer</h4> So, this idea is very powerful and much deeper than just "do rebases". The real idea here is "test before you merge." It just so happens that rebasing, with few enough engineers sharing a repository, tends to be an easy way to achieve this. Regardless of how you eliminate three way merges, these kind of tactics make your earlier branch tests more effective, making your existing tests work earlier. Solution 3. Test Outside The Box</h3> Unit and integration tests that I've written and seen typically work inside the same process as the application code under test. Mocking is used to speed up expensive operations and create very specific situations to constrain tests. All of that is great at testing very specific code paths efficiently, and it gets us something like 80% of the correctness guarrantees with 20% of the work of, say, fuzzing. So, this is really great stuff and its awesome we do it. But, it doesn't really answer if the application actually works for users. I like to get paid. I like it when users like the software I write and they pay for it...you know, because it worked for them. Life is hard enough without glitchy software, why make people suffer more? By buttressing our "whitebox" testing with tests that operate outside the application as a separate process, we can check detect serious regressions. Additionally, these tests are very portable. With a tinsy bit of cleverness, you can write blackbox tests to run against a remote machine just as easier as a separate process on your same machine. Cons with Blackbox Testing</h4> Upfront investment in blackbox testing can discourage managers. However, I find that writing blackbox tests during a proof-of-concept phase to be worthwhile effort. Because the tests are portable and separate from the application, I can throw away my prototype and keep the same tests. If I've written enough coverage of my application using blackbox testing, I can be sure that any services using my application can switch over to the new application flawlessly. I can even write the application in a different language my blackbox test! Maintenance of blackbox tests can intimidating. My prefered metholodogy would be make the blackbox tests mandatory for CI, and encourage developers to add blackbox tests alongside new features. This does impact development velocity, but in many cases the existing tests can remain unchanged. Pros with BlackBox Testing</h4> You know those annoying bits of code to test near your program entrance? Using blackbox testing with code coverage</a> tools can make it easy to see that main method and many other wide swathes of your application have abundant test coverage. Again, because blackbox tests can be written for portability, you can run them against production like environments from your own machine or from CI and experiment with changes before you get to production...if you only had your own production like environment... Conclusion</h2> Each of my three solutions can work separately. But, there's magic that happens if you put them together: if code can only be fast-forward merged, and is tested on your development branch from a staging environment, there's no need to run the test suite again on master...until after you deploy to production! In my experience, this could easily 30% of your deployment time that was previously dedicated to re-running tests. Automate Lets Encrypt SSL Certificate Management with Gitlab Pages 2019-07-21T00:00:00+00:00 Let's Encrypt offers free automated SSL certificate issuance, and its become the prefered way to provision and renew SSL certificates nearly overnight since its debut a few years back. Gitlab Pages (where I'm hosting this blog) encourages users to rely on Let's Encrypt but doesn't document an automated way to manage this. Why Automate Certificate Renewal?</h2> Let's Encrypt certificates are only valid for 90 days, which is just often enough to require some effort but too infrequent to ever remember how to do this. Also, the stakes are kinda high: if I don't keep SSL working, Google will demote my site in search rankings and visitors who bookmarked and/or typed in my URL directly will see something horrible like this: How Can We Automate Renewal of SSL Certificates for Gitlab and Let's Encrypt?</h2> Isolate _acme.<YOUR HOST NAME></code> in a hosted zone record set with Route53</li> Provision an IAM user with limited permissions to control this hosted zone's record sets.</li> Figure out the certbot-dns-route53 and Gitlab Pages API command to obtain a certificate</li> Schedule a recurring Gitlab CI to renew regularly</li> Monitor using Keybase.io and/or uptimerobot.com to discover if this stops working.</li> Profit!</li> </ol> The main reason for (1) is to separate privileges so that if a hacker obtained the credentials for Route53, they couldn't pwn ALL your DNS; they could only replace your SSL certificate for the one domain. This is discussed at length elsewhere. AWS is one of the few DNS providers that provide granular access control for DNS. In 2019. The secondary reason is so that we can leverage certbot</a> with the certbot-dns-route53</a>. Previously, I've always used HTTP01 challenges, which require putting a random string LetsEncrypt at a specific path on my site, which worked well when I was directly managing an S3 bucket. However for Gitlab Pages, I can only puth content at a path by pushing commits to Gitlab, but I really don't want to do that to my git history. The DNS01 challenge offers an alternative which is orthogonal to git history. (2) Tests our ability to use all these things together and check the credentials for everything. (3) Once we've shown that the script works, we just schedule to run automatically. (4) The whole point of this setup is to avoid doing a lot of manual work. If you have to visit your website every N days and read through the very confusing certbot documentation, you'll forget and then something will happen and you won't know you're down. DNS Setup with Route53</h2> Create your Route53 hosted zone with the domain name _acme-challenge.<DOMAIN NAME FOR SSL></code>. So for this blog, the domain for Route53 hosted zone is _acme-challenge.bryce.fisher-fleig.org</code>. Look for the alphanumeric hostzone ID in the AWS console and remember this for the next step. If like me, you don't use Route53 to manage all your DNS, you'll need to make sure that some authoritative records point to the Route53 hosted zone. In my case, I purchased fisher-fleig.org</code> from GoDaddy before Route53 was even an AWS product. So, I had to add records pointing to _acme-challenge.bryce.fisher-fleig.org</code>. I choose to use a "name server" NS</code> record, but you could easily use a CNAME</code> record as well. Let's verify that this works by using dig</code>: $ dig -t NS _acme-challenge.bryce.fisher-fleig.org ;; ANSWER SECTION: _acme-challenge.bryce.fisher-fleig.org. 588 IN NS ns-1800.awsdns-33.co.uk. _acme-challenge.bryce.fisher-fleig.org. 588 IN NS ns-24.awsdns-03.com. _acme-challenge.bryce.fisher-fleig.org. 588 IN NS ns-834.awsdns-40.net. _acme-challenge.bryce.fisher-fleig.org. 588 IN NS ns-1495.awsdns-58.org. </code></pre> I left out a lot of the dig output not revelant to this, but the "ANSWER SECTION" shows that several domains like .awsdns-</code> are acting as name servers for _acme-challenge.bryce.fisher-fleig.org</code>. When you provision a hosted zone in AWS, you'll be provided your own list of name servers that are likely different from mine. You'll need to consult with your DNS provider's documentation for how to setup the NS records for the AWS name servers specific to your hosted zone. Using my name servers likely won't work for your domain. Regardless of where you host your DNS for the domain you want to secure, you should be able to add a TXT record (for example with "hi mom") inside Route53 for _acme-challenge.<YOUR DOMAIN></code> and then see that same way in dig a little later: dig -t TXT _acme-challenge.<YOUR DOMAIN> ;; ANSWER SECTION _acme-challenge.<YOUR DOMAIN>. 600 IN TXT "hi mom" </code></pre> If you can see "hi mom" or whatever value you set, then it works! Move on to the next step. IAM Setup to Manage Route53</h2> Now, create an AWS IAM user with this inline IAM policy: { "Version": "2012-10-17", "Id": "certbot-dns-route53-bryce-fisher-fleig-org", "Statement": [ { "Effect": "Allow", "Action": [ "route53:ListHostedZones", "route53:GetChange" ], "Resource": [ "" ] }, { "Effect" : "Allow", "Action" : [ "route53:ChangeResourceRecordSets" ], "Resource" : [ "arn:aws:route53:::hostedzone/<HOSTED ZONE ID>" ] } ] } </code></pre> Name the policy something and add it to your user. Store the AWS_ACCESS_KEY_ID</code> and AWS_SECRET_ACCESS_KEY</code> for later. Figure out Certbot and Gitlab Pages API</h2> I spent roughly an hour trying to get the official Pypi certbot-dns-route53 package</a> to work for me locally in a virtualenv on MacOS, and didn't have it working. I found the official Docker image</a> to be much easier to setup and use. Plus, this image can be used with very few tweaks from Gitlab CI. Ask for a Certificate</h3> export AWS_ACCESS_KEY_ID=$ACCESS export AWS_SECRET_ACCESS_KEY=$SECRET certbot certonly \ -n --agree-tos \ --email "$EMAIL" \ --dns-route53 \ -d "$DOMAIN" </code></pre> The plugin requires AWS credentials to be passed as environment variables</li> certonly</code> means that certbot won't try to install the certificate anyway, it will just output the certificate to a file</li> $EMAIL</code> is your email address (required, but also for notices about certificate expiration)</li> $DOMAIN</code> is the domain you want a certificate for</li> </ul> If that all works, you should now have the secret key in /etc/letsencrypt/live/${DOMAIN}/privkey.pem</code> and the full certificate chain in /etc/letsencrypt/live/${DOMAIN}/fullchain.pem</code>. Obtain an API Token for Gitlab</h3> Go to https://gitlab.com/profile/personal_access_tokens</a> and check the box for Scopes labeled "api". This token does seem more permissive than perhaps it should. If you want to lock down this down more, you could create a dedicated gitlab user for only your blog instead of all your projects. Send the certificate to Gitlab Pages</h3> curl -XPUT \ -H "PRIVATE-TOKEN: $GITLAB_TOKEN" \ --form "certificate=$(cat "/etc/letsencrypt/live/${DOMAIN}/fullchain.pem")" \ --form "key=$(cat "/etc/letsencrypt/live/${DOMAIN}/privkey.pem")" \ "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pages/domains/${DOMAIN}" </code></pre> $CI_PROJECT_ID</code> is the unique number Gitlab assigns to each project. For example, this blog has a project ID of 2085659 and you can find it on the main Gitlap project page, just under the title. Automate All The Things!</h2> You should be feeling pretty good right now -- you've written up a script that hands a shiny new SSL certificate any time you want. Nice! Now, lets make it happen automatically by setting a scheduled job in Gitlab CI. Setup the GitlabCI Yaml</h3> I spent most of my time figuring out the right incantion for my .gitlab-ci.yml</a>: letsencrypt: only: - schedules image: name: certbot/dns-route53 entrypoint: [""] script: - apk add curl - 'certbot certonly -n --agree-tos --email "${EMAIL}" --dns-route53 -d "${DOMAIN}"' - 'curl -XPUT -H "PRIVATE-TOKEN: $GITLAB_TOKEN" --form "certificate=$(cat "/etc/letsencrypt/live/${DOMAIN}/fullchain.pem")" --form "key=$(cat "/etc/letsencrypt/live/${DOMAIN}/privkey.pem")" "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/pages/domains/${DOMAIN}"' </code></pre> entrypoint: [""]</code> -- this was very difficult to figure out, even with the documentation</a> in hand. My guess would be that Gitlab Runners actually use docker exec</code> to run script</code> commands on a container, but many containers (like certbot/dns-route53</code>) will exit immediately if not given a sensible command for docker run</code>, though to be honest I'm not quite sure how this works.</li> only: schedules</code> -- this is important to prevent every push to your repo from generating a new SSL certificate, but it means you'll have to visit the schedules page and click the play button to actually your code</li> apk add curl</code> -- the docker image doesn't come with curl pre-installed so we leverage Alpine Linux's package manager apk</code> to install it</li> </ul> Put It on Your Schedule</h3> Gitlab CI schedules are controlled separately from .gitlab-ci.yml</code> via the API and using the website: I do wish that schedules could be created via the regular .gitlab-ci.yml</code> file used for everything else in Gitlab CI...but at least there's an API and a fairly straightforward UI for this. The upshot of separating out the scheduled jobs is that credentials for scheduled jobs can be managed separately from git-push jobs. We can benefit from that by only adding the AWS_ACCESS_KEY_ID</code> into the scheduled job for Lets Encrypt (and not in the regular variables used by git push). Pick the right branch here (I used master once I'd finished manually verifying everything works). These variables will need to be setup for the schedule: EMAIL</code> - your email address</li> DOMAIN</code> - the domain you want to secure with SSL</li> AWS_ACCESS_KEY_ID</code> / AWS_SECRET_ACCESS_KEY</code> - you should have this stored away from step 1 above</li> GITLAB_TOKEN</code> - this token you created inside Gitlab in the previous step</li> </ul> Once its all added, test it out by click "Play" on the schedules overview page and watch the job logs to look for problems. Monitor on the Cheap</h2> I've used UptimeRobot</a> for several years and I'm pretty happy with their free offering for personal projects. However, I've found that Keybase</a> (which is for encrypted chat and file sharing, not monitoring things) is actually really good at noticing when my SSL certificate is expired on my blog...and its also free. I'm not going to go into how to use these products since you've suffered through enough of my yammerings already, but I think either one can be a great option. Profit!</h2> Using this system, you should have a perpetually valid SSL certificate, replaced at the frequency of your choosing, and costing you a few cents per month on AWS Route53 bills. Dangers of CloudFront for Small Sites 2019-07-05T00:00:00+00:00 For many years, I've relied on CloudFront to provide a professional level of availability for pennies a month. However, for the last two months, I had to take this blog offline to protect my wallet. History of my Blog's Tech Stack</h2> When I first started blogging about 6 years ago, I was a passionate Drupal developer, and I built my content on a simple Drupal site. Fast forward to 2014, I learned about AWS S3 and did a fair bit of work using CloudFront for some Drupal sites I running at work. Once I learned about Jekyll, I rebuilt my blog using Jekyll and used the aws cli to push that content to S3 where CloudFront served the content over HTTPS with a custom SSL certificate</a>. Running a production website without managing servers is what made me fall in love with cloud computing. Eventually, I figured out how to use Travis CI to automate pushing new content to S3, migrated to Gitlab/CI, and added the ability to have each branch of my blog create a new S3 bucket for me to preview the content. I added simple bash scripts to garbage collect old buckets, and do all of the tasks I felt needed it. Then, last week, I gave up all of the power and control I had over this blog from managing CloudFront myself and ruefully switched to Gitlab Pages. What Happened To My Blog on CloudFront</h2> Yes, the scale on this graph is from $0 to $2 per month. Up until March of 2019, I'd never spent more than $2 on my whole AWS account in a single month. Then, a funny happened in April: When I got this bill in May, I immediately turned off the CloudFront distribution and slowly investigated what went wrong. Where did these costs come from?</h2> I discovered IP addresses based in Turkey were making HTTP HEAD requests for /ping.txt</code> (which has never existed). I've lost the report from CloudFront (since the reports have a 3 month retention period), but the Cost Explorer and pricing information shows: $0.012 per 10,000 requests</li> $150 total costs</li> </ul> So, $150 10,000 requests / $0.012 = 125 million requests, or roughly 1 request per second. I thought that there were more like 15 billion requests, but the math does not check out for that. If my memory is correct, then CloudFront serviced close to 5000 requests per second for a month. Why Didn't I Notice This Earlier?</h2> I didn't setup billing alerts with AWS. If you're running personal infrastructure on AWS and don't have billing alerts, stop what you're doing and go setup some billing alerts right now. The upshot of all this is that CloudFront kept my blog online and available during a spike in traffic of many orders of magnitude. I didn't have any downtime detected by my monitoring tools. But the downside of this uptime is that I literally paid with my wallet. I'd gladly have traded some downtime (like weeks of downtime) to have that $150 back in my pocket. How Can I Protect My Blog?</h2> This is the really frustrating part for me. There really weren't any great options that I could find. Here's what looked into: Geo-blocking inside CloudFront</h3> CloudFront can be configured to block access to content based on the presumed physical location of visitor's IP address. The problem is you still pay for the geo-blocked requests. The point of geo-blocking is to (a) protect your original and (b) limit access to your content. Geo-blocking isn't designed to really save you from an attacker. WAF and DDOS Protection</h3> Like CloudFront geo-blocking, all that Web Application Firewall (WAF) and the various DDoS protection products try to do is restrict access to the origin, not prevent CDN costs. DNS to the Rescue!</h3> So, what we learned above is basically if a request goes to CloudFront, you must pay for that request. Full stop. But, what if we could selectively control which requests went to a CloudFront? What if we could use DNS to deny access to CloudFront to the attacker? At work, we use a product called Cedexis which is just all kinds of amazing. Basically, its a programmable DNS resolver. You define several "platforms" which are CNAMEs that Cedexis could resolve, and then you feed analytics into Cedexis, and at DNS resolution, custom javascript applications have access to the analytics data to let pick different platforms based on which one is most available or performant or cheap. Unfortunately, Cedexis is only really marketed as an enterprise product, not a hobby developer product. Fortunately, AWS Route53 has DNS policies that can do basic geo-blocking (and probably a lot more too). Sadly, each policy costs $50/month, so that doesn't really help me keep costs down to under $2. Ditch CDNs for VMs</h3> This is actually a pretty reasonable idea. Getting a fixed $5/month cost and variable uptime is exactly the set of tradeoffs I'm looking for. Also, it gives me considerable flexibility to build out automation for previewing different branches of my git repository for this blog. The downsides are that I'd only be willing to pay for one VM running in a single location, so pagespeed would likely vary based on the visitor's distance from my VM, and I do get some percentage of my visitors from the UK and India. Also, I'd be committing to spending $5/month regardless of whether the server was idle or active during that time. Also, now instead of sitting back and letting CloudFront provision and monitor individual servers, I'd have to setup monitoring and alerting for this VM. DIY Cedexis</h3> I actually want to do this. After looking at BIND and CoreDNS, I didn't see anything quite as powerful as what Cedexis OpenMix provides, but I did find this golang library for DNS</a> that's a promising start. My main concerns here are about downtime and abuse of the DNS resolver...but I plan to experiment in this area. If I get this right, I could potentially keep using CloudFront + S3 + LetsEncrypt to manage this website. The endgame for this would be to automate attack detection and mitigation as a sort of fail2ban at DNS layer. Gitlab Pages</h2> I wound up going with Gitlab Pages for now because my time is so limited and I wanted to be back online (after weeks of downtime) as soon as possible. The biggest things I lose here are: easy way to renew LetsEncrypt certificates</li> branch previews of my markdown</li> visibility into whats going on with the delivery of my blog</li> control over custom redirects, protocols, etc</li> surprise bills for $150</li> </ul> Parting Thoughts</h2> If you know of other ways to financially mitigate even simple attacks while keeping CloudFront in my stack under my control, please do tell me! Also, billing alerts are key. Separate IO from Logic 2018-05-17T00:00:00+00:00 I've come across some terrible code at work lately. It was written by me a few months earlier. When I tried to think about why it was so bad, I realized that I hadn't separated my business logic from all the "glue code" to other network services. Once I recognized this anti-pattern, I started seeing it lots of places. What Is This Terrible Code?</h2> Here's something very similar to the actual code: def autoscale(): all_jobs = ORM.query_for_unfinished_jobs() total_work = 0 for job in all_jobs: if job.is_valid(): total_work = job.cost time_sla = lookup_time_sla() worker_speed = lookup_worker_speed() ideal_worker_count = round(total_work / (worker_speed * time_sla)) min_workers, max_workers, actual_worker_count = AWS.describe_auto_scaling_group() if ideal_worker_count > max_workers: ideal_worker_count = max_workers if ideal_worker_count < min_workers: ideal_worker_count = min_workers if ideal_worker_count > actual_worker_count: PAAS.scale_up_to(ideal_worker_count) elif ideal_worker_count < actual_worker_count: PAAS.scale_down_to(ideal_worker_count) </code></pre> The actual code has a few additional constraints and lots and lots of metric-spewing instrumentation so we can see what its thinking...all of which makes the code that much harder to reason about and all the more important to test! What Does This Code Do?</h3> Essentially it does some math to decide how many servers we need to complete our long running batch jobs within a specific number of seconds (the "SLA"). Here's the essential formula: number of workers = combined time estimates for all jobs / (speed of one worker * the SLA) </blockquote> Let's mull over how changing the value of each individual variable in that formula effects the number of workers: As the total amount of work increases (all else being constant), we'll need more workers</li> As the speed of the workers increases, we'll need fewer workers</li> If we want the work to get done faster (by decreasing the SLA), we'll need more workers</li> </ul> So far so good! Those outcomes all seem reasonable and intuitive. The gist of the formula feels right. Oh, the Edge Cases!</h3> BUutttttt -- there's a whole lot of additional concerns we need to think about to ensure this code is getting us to the correct outcomes. For one thing, some jobs might be invalid, so we'd want to exclude them from the calculation. For another, scaling up and scaling down work totally differently because we want a graceful shutdown, not interrupting a job in progress. Finally, we set boundaries on the min and max sizes of our pool of workers. If the ideal number of workers is more than the limit, we need to respect the limit. (In real life, there's at least three other constraints we need to adhere to as well.) If we get this formula or its edge cases wrong, we're either wasting money or making our customers frustrated. Its safe to say the stakes are high! Testing This Is Terrible</h2> So, how do I test the scale up and scale down behavior in various scenarios? There's no arguments to this function and there's no return values. However, you'll notice that there's basically one main mathy thing going on here to calculate the ideal_worker_count</code>. Let's the consider the case of a cold start. I have no active workers, and suddenly a whole slew of jobs come in at once. Let's check (1) we scale up and (2) we scale up to the correct amount. Given the above implementation, it would look something like this: @patch.object(PAAS, 'scale_up_to') @patch.object(PAAS, 'scale_down_to') @patch('path.to.code.lookup_time_sla', return_value=300) @patch('path.to.code.lookup_worker_speed', return_value=2) @patch.object(AWS, 'describe_auto_scaling_group') @patch.object(ORM, 'query_for_unfinished_jobs') def test_scale_up(m_query_for_unfinished_jobs, m_describe_auto_scaling_group, m_lookup_worker_speed, m_lookup_time_sla, m_scale_down_to, m_scale_up_to): job = Mock(cost=120) job.is_valid.return_value = True m_query_for_unfinished_jobs.return_value = [job for _ in range(0, 10)] m_describe_auto_scaling_group.return_value = (0, 10, 0) autoscale() # System under test m_scale_up_to.assert_called_once_with(2) </code></pre> This test stinks. Its so counterintuitive, we literally wrote out comments with the math spelled out as grade-school math homework. While I'm quite happy to mock out lots of things in my unit tests like this, how each of the different mocks interacts to reach that result of 2</code> at the bottom is extremely non-obvious. Now imagine around 10 different tests that are variations of this code for the various scenarios and intuitions I've outline for this piece of code. Good luck trying to debug this or extend it in the future! There Must Be a Better Way</h2> Let's step back a second and think about what we're really trying to do here in the test: We want to know that given some set of jobs and some set of workers, we arrive at the correct final size. It's almost like all the various bits of information should be arguments to a function, and the size of the group should be the return value. Let's forget about where all the data comes from and let's focus on seeing what a function would look like that just did some math instead of finding all that data. Here goes: def autoscale_math(min_workers, max_workers, total_work, worker_speed, time_sla): ideal_worker_count = round(total_work / (worker_speed * time_sla)) if ideal_worker_count > max_workers: return max_workers if ideal_worker_count < min_workers: return min_workers return ideal_worker_count </code></pre> This code almost feels too terse to qualify for being its own separate function. But, how do the tests look for this? # Let's keep these bits constant for our test MIN_WORKERS = 0 MAX_WORKERS = 10 WORKER_SPEED = 2 TIME_SLA = 300 def test_scale_up(): current_workers = 0 total_work = 1200 ideal_worker_count = autoscale_math(MIN_WORKERS, MAX_WORKERS, total_work, WORKER_SPEED, TIME_SLA) assert ideal_worker_count == 2 </code></pre> Already, this seems quite a bit clearer for testing as well as for the application code itself. But, if we want to really cover a significant portion of the different cases, perhaps we could be even more expressive somehow. What if we used test tables to layout all the different test cases? Let's see how that would look in python: MIN_WORKERS = 0 MAX_WORKERS = 10 WORKER_SPEED = 2 TIME_SLA = 300 test_autoscale_cases(): test_cases = [ # total_work | ideal_worker_count | description ( 1200, 2, 'cold scale up'), ( 3000, 5, 'scale up partially'), ( 1000000, 10, 'only scale up to max'), ( 0, 0, 'scale down totally'), ( 300, 1, 'scale down partially'), ( -1000000, 0, 'only scale down to min'), ] for (total_work, ideal_worker_count, description) in test_cases: assert autoscale_math(MIN_WORKERS, MAX_WORKERS, total_work, WORKER_SPEED, TIME_SLA) == ideal_worker_count, description </code></pre> This style of testing is considered idiomatic in golang</a> and is also my favorite feature of Gherkins</a>. I don't see a lot of python projects using this style, but it definitely works toward the overarching goal of most python syntax: readability. The test tables allow me to think only about the specific case I want to test without thinking about the boilerplate to drive the test into that state. Combining that style of test with a very simple, pure function</a> results in super easy test coverage for critical systems like this. Notice that my test table doesn't include coverage of the invalid jobs -- that's okay! I've managed to compress so much boilerplate by using this test table, that I'll just add another one-off test for the invalid job edge case. Trying to shoe-horn that bit into the above table wouldn't really make sense to me because I'd have to add a column to every row, but really invalid jobs would only effect one of the tests. But About All That Glue Code?</h3> Right! So lets' try breaking about all the network calls into its own function and see if that makes any sense: def autoscale(): time_sla = lookup_time_sla() worker_speed = lookup_worker_speed() min_workers, max_workers, current_workers = AWS.describe_auto_scaling_group() jobs = ORM.query_for_unfinished_jobs() total_work = [job.cost for job in jobs if job.is_valid()] ideal_worker_count = autoscale_math(current_workers, min_workers, max_workers, total_work, worker_speed, time_sla) if ideal_worker_count > current_workers PAAS.scale_up_to(ideal_worker_count) elif ideal_worker_count < current_workers PAAS.scale_down_to(ideal_worker_count) </code></pre> This function has also become dramatically simpler. Looking at this function, its clear that we also need to think about the failure modes for these calls as well as the business logic. For instance, should we retry the PAAS.scale_up_to()</code> calls if they fail at first? Do we want a hardocded fallback for lookup_worker_speed()</code>? I'll leave those details as an exercise to the reader as well. When Separating Logic and IO Does Make Sense</h2> The key feature that made separating the IO and logic work for this case was that we want all the data from various network calls every time. As long as we need to always gather the same data every time (or doing that IO is considered cheap in your context), then we don't have to be strategic about when to gather the data. We just always get all of it. When Separating Logic and IO Doesn't Make Sense</h2> Sometimes, various networks calls which are input to your buisness logic can and should be skipped. In those cases, the refactor technique only makes complex code harder to read. If we try to pull all of the IO out but still be strategic, now we end up making lots of little functions to evaluate lazily. If you strongly prefer very short and numerous 1-2 lines functions over 100-200 line functions, maybe separating the IO and Logic still makes sense. However, when I've tried this, my coworkers didn't thank me and there was just a lot more code to try to understand. Additionally, to make the network calls lazy, I had all more functions which just added more boilerplate to sift through. The end result was greater head scratching. For these cases, there's a number of refactor strategies you could try: Look for dead code that can be totally removed</li> Simplify your data model to reduce the differences in these various scenarioes</li> Use some kind of caching to make the data less expensive to look up</li> Consider breaking apart a series of smaller chunks of IO + logic units and chain them together</li> Try using a cucmber clone to write narrative descriptive tests around the icky bits</li> What component has too many responsibilites? Can I split out one muddled abstraction into two more narrow abstractions?</li> </ul> Sometimes, the nasty code refuses to yield to your gentle coaxing, and that's okay too. Conclusion</h2> Sometimes, separating your core logic out from IO can greatly enhance the readability and testability of your code. Try using test tables to greatly reduce amount of boilerplate you need to write. Also, make sure that your code always fetches the same data before trying to isolate logic and network concerns. Recurse Center Day 5: Wrapping Up 2018-01-12T00:00:00+00:00 I want to learn more about how video codecs work so someday I can grow up to be a Jedi-master Video Engineer. To aid me on this epic quest, I'm doing the one week "mini-batch" at the Recurse Center</a>. Here's my notes from Day 5. Late Night Reflections</h2> Last night, I was chatting with three other programmers from RC and Geoffrey said somethings that got lodged in my head: Athletes with no practice</h3> Can you imagine being an athlete that never practiced and only played real games? And yet that's what we do in software development. We never practice the basic skills. </blockquote> Then he recounted how one of his friends was practicing how to quickly setup a new angular app from scratch. The friend would schedule 90 minutes to do "reps", and give himself 15 minutes to go from 0 code to a running angular app. At the end of the 15 minutes, he delete the app and start again. Then he'd stop after the 90 minutes were over. Geoffrey said, "Why don't practice code review and team communication and other skills too? I think I'm going to write a new lisp interpreter every three months to keep practising." I feel like this was one of those watershed moments where the way I think about how to improve has radically altered. Failing to learn from failure</h3> One last profound thought from Geoffrey last night: Failing to learn from failure is the only unforgivable failure. </blockquote> While I totally agree with that sentiment, I feel like that's a thought I need to chew on and try to internalize. Reflecting on my own decisions and actions isn't something that comes naturally, but it feels like a skill I ought to focus on. Pair Blogging</h3> While chatting with these recursers, I came up with the idea of "pair blogging": it's pair programming except that the "navigator" is responsible to live blog what the pair is programming, code samples, screenshots, problems, design decisions, links to relevant documentation etc. Then when the pair switch roles, the other person writes "Ok, Bob writing now..." and the other person takes over writing. Not sure if it's a terrible idea or an amazing idea... Setting Intentions</h2> Today, I'm going to practice timeboxing work. I'll write a repl explorer for an mkv file in 30 minutes, three times (doing "reps"), and then after lunch I'll do other projects: timebox trying to compile to web assembly to an hour, and do three half-hour reps of using bindgen to make rust bindings for libvpx. Doing Reps - Making an mkv repl</h2> Doing this project 3 times in a row was just amazing. I never quite got it working all the way, but I've gotten much better at making the repl itself. The difference between rep 1 and rep 2 as immense. I was able to really get good at making the repl itself with multiple commands in an idiomatic Rust fashion. I'm really proud of my work in rep 2. At the end of rep 2, I had barely started figuring out how to interactively traverse the tree of EMBL Elements, so rep 3 (which was only 15 min) focused on that. I'll need several more reps to figure it out... The whole ownership / Rc</code> data structure got in the way. I was fighting the borrow checker the whole. I think, fundamentally I just don't understand how Rc</code>'s work at all -- which is really embarrassing after having so much Rust for fun. See the code from all three mkv-repl reps on gitlab</a>. Honestly, 3 reps was not enough to have making a repl down pat, let alone figure out how to navigate the tree while appeasing the borrow checker. I really want to do this more in the future. Having a wife and kids who I want and need to spend most of my nonwork hours with, one of the main frustrations I face on a regular basis is that there's nothing meaningful I can do with the scraps of 30 minute blocks of time I have. Now, with reps I have a new way to feel good about doing a mini project on a regular basis that will let me improve and fit into the time I actually have available. Compiling Rust to WebAssembly (wasm)</h2> Okay, I've got one hour...where do I even start here? This is so crazy! I remember there was blog from Rust lang</a> about doing recently. Nope, I can't see any posts about this.. Let's duckduckgo for it. There's a post for Rust 1.14 back in 2016...no way it was that long ago. Uh, yep it was that long ago. Let's find the really really basic example</a> of how to do this from the Rust team(s): Rust installation:</h4> rustup target add wasm32-unknown-emscripten </code></pre> If you already have rustup installed, then instead of running the curl and source commands above, just switch to the nightly toolchain with rustup default nightly. Emscripten installation:</h4> curl -O https://s3.amazonaws.com/mozilla-games/emscripten/releases/emsdk-portable.tar.gz tar -xzf emsdk-portable.tar.gz source emsdk_portable/emsdk_env.sh emsdk update emsdk install sdk-incoming-64bit emsdk activate sdk-incoming-64bit </code></pre></blockquote> I have rust already installed, I added toolchain...do I really need to install emscripten separately? Let's just try compiling to the wasm target without installing emscripten: echo 'fn main() { println!("Hello, Emscripten!"); }' > hello.rs rustc --target=asmjs-unknown-emscripten hello.rs </code></pre> No love. Under the hood, rustc</code> is shelling out to emcc</code> (presumably the emscripten compiler) which is not found on my system. Ugh, okay, actually install empscripten. Step 1, curl a thing - check. Step 2, untar a thing - check. Step 3, source a thing - nope. There's a typo -- it should be source emsdk-portable/emsdk_env.sh</code>. Check. Step 4, update a think, check. Step 4, install the 64 bit sdk, check.....nope, this taking forever. Omg, its still going. Still going. I'm only 55% done and its been running for approximately 35 minutes. Not sure this is going to work out...good thing I have a blog to finish up. It feels like I should be doing something to help to actually write the code and compile to wasm while I wait...but I don't want to forget to watch for the compilation to finish or I'll not get anywhere in the 19:05 let on my timer. Let's lookup how to set the target</a> for a project in the Cargo.toml file. Arg, I can't find anything about how to tell cargo which platform to target. Let's check the cargo help</code> command...Great! cargo build --help</code> tells that the magic flag is --target</code> and I need to put the target triple after that. I know rustup toolchain list</code> can tell what the target triple is for wasm...just kidding rustup target list</code> it is. The target is wasm32-unknown-emscripten</code>...which is what I copied from the rust user forum above. Perfect! 78% of emscripten is build...only 10 minutes left for my hour time box... What else can I do? I supposed I could start writing something that wraps the mkv crate, but if the hello world code doesn't compile whats the point? 79% compiled.... 80% compiled... Hmm, I guess I could look at code samples that suspected to work with wasm...Okay, here's this great post about baby steps in Rust -> wasm</a> which is the same as I've done (but much better written). It took him 2hrs to compile...so either my compilation is actually very fast, or there's whole other pass after the 81% finished job is done. Anyway, the author suggests enabling a config flag in firefox...let's do that! Here's my about:config default settings: 84% compiled...less than 1 minute to go Okay, I'll conceded my time is up. Oh well! I knew it was a long shot. Next time I want to try this, I'll have emscripten all ready to go (hopefully!). Recurse Center Day 4: No, really I am going to decode a frame 2018-01-11T00:00:00+00:00 I want to learn more about how video codecs work so someday I can grow up to be a Jedi-master Video Engineer. To aid me on this epic quest, I'm doing the one week "mini-batch" at the Recurse Center</a>. Here's my notes from Day 4. Late Night Brainstorming</h2> So, on my walk home last night, I had a whole flood of new ideas for this project come into my head. Here's a quick overview of some of the most interesting ones: Make an interactively GDB-style repl for exploring the elements in a Matroska container - I could almost definitely do this in a few hours</li> Compile to WebAssembly and create a client-side Matroska parser that can: present a tree explorer using [+] and [-] buttons, like a bit like Finder</li> create a visualization of frame size along a timeline, possibly passing the same data to a video element and seeking to the frame on hover</li> decode vpx frames, analyze frame data and visualize them and show them in a canvas</li> </ul> </li> Some mashup of these things</li> </ul> Setting Intentions for the Day</h2> Okay, so I'm committing to digging into decoding a frame. I feel like if I'm willing to properly ask for help, so many more interesting things will be unlocked for me. Plus...I've told literally everyone that's what I want to do! After I finish that (and maybe only that today), then I'll try compiling to WebAssembly and doing fun things in the browser. A few duckduckduckgo's later, I find this super helpful looking header file in bitstream guide</a>. Hmm, after reading that it still seems like I want to be looking at simple_decoder.c to grok libvpx and perhaps just use the header file as a bit of documentation about the data structures. My goal is to simply rewrite simple_decoder.c so that I can just feed it a keyframe (I'll figure out how to get just one keyframe later on). Rewriting simple_decoder.c in C</h2> Right now, I'm totally buggered by all the video_reader_* stuff coming out of video_reader.c. Rewriting needs to entirely replace that code. 0 - Can I recompile it?</h3> Since this is stored in git, I can just edit that file. Let's see make sure really do know how to recompile this file. I'll add a gratuitious print statement at the beginning and just try to run that. I add a die("Hello!")</code> as the first line in main()</code>, and then recompile it like this: $ cd build $ make make[1]: Nothing to be done for `all'. [DEP] examples/simple_decoder.c.d [CC] examples/simple_decoder.c.o [LD] examples/simple_decoder make[1]: Nothing to be done for `all'. make[1]: Nothing to be done for `all'. </code></pre> Great! Make was smart enough to only recompiled simple_decoder for me. Fantastic! Let's see it work: $ ./examples/simple_decoder Hello! Usage: (null) <infile> <outfile> </code></pre> That's odd -- well, my code clearly ran, but the program didn't run the way I expected at all. I thought that die()</code> would actually stop execution, but we still saw the usage info code being run, so that just proves I don't know C. 1 - Figure out video_reader is doing</h3> The main loop of the program looks like this: while (vpx_video_reader_read_frame(reader)) { vpx_codec_iter_t iter = NULL; vpx_image_t img = NULL; size_t frame_size = 0; const unsigned char frame = vpx_video_reader_get_frame(reader, &frame_size); if (vpx_codec_decode(&codec, frame, (unsigned int)frame_size, NULL, 0)) die_codec(&codec, "Failed to decode frame."); while ((img = vpx_codec_get_frame(&codec, &iter)) != NULL) { vpx_img_write(img, outfile); ++frame_cnt; } } </code></pre> There's two calls to video_reader_</code> things in this part: while (vpx_video_reader_read_frame(reader)) {</code> - just from squinting at this code, it looks to me like the reader</code> contains all the state, and vpx_video_reader_read_frame</code> must be returning false or NULL whenever reader</code> has reached the end, possibly returning something truthy before then.</li> const unsigned char frame = vpx_video_reader_get_frame(reader, &frame_size);</code> - a few observations here:</li> </ol> frame_size</code> is initialized as 0, but apparently is nonzero later on. So, it must be passed by reference so that vpx_video_reader_get_frame</code> can mutate it...presumably because multiple return values is hard in C</li> frame</code> holds all the coded data</li> </ul> So, theoretically, if I could hardcode the values of frame_size</code> and frame</code>, I could remove all the video_reader_</code> things. Great! Let's try printing those things out in the shell on the first iteration, crash the program. So the main loop becomes: while (vpx_video_reader_read_frame(reader)) { vpx_codec_iter_t iter = NULL; vpx_image_t img = NULL; size_t frame_size = 0; const unsigned char frame = vpx_video_reader_get_frame(reader, &frame_size); printf("frame_size=%d", frame_size); // THIS IS PRINT STATEMENT! if (vpx_codec_decode(&codec, frame, (unsigned int)frame_size, NULL, 0)) die_codec(&codec, "Failed to decode frame."); while ((img = vpx_codec_get_frame(&codec, &iter)) != NULL) { vpx_img_write(img, outfile); ++frame_cnt; } } </code></pre> Now the compiler complains that my format string is wrong...and has helpfully suggested replacing %d</code> with %zu</code> (whatever that means!). Great! Now I can frame_size being printed. Let's try that again for the frame</code> variable too...apparently the right format string is %s</code> (I actually know that one from golang and python!). Also, I can't imagine that string is the right way to visualize encoded binary data...and it's not. I see nasty unicode things in the terminal (but it did compile). Let's actually stop and learn about C-style format strings. A quick skim through man printf</code> shows that there's a %b</code> format which is like %s</code> but escapes differently (including octal) -- but that doesn't produce the output I want. There's also a %X</code> for octal but it only prints the first octet, not the entirety of frame</code>. No idea how to do this, time to duckduckgo how to print out binary data from C in printf. So, it looks like we can iterate over frame and print each value as %x</a>: printf("\nframe_size=%zu | frame=\n", frame_size); for (int i = 0; i < frame_size; i++) { printf("%02x", frame[i]); } </code></pre> Amazing! So I wind up with things like this as output: frame_size=1650 | frame= 114d001d10e4147b8c4fd3c78f2713ff7daac8.... </blockquote> Now, I need to figure out how I would actually create a value I would copy and paste into a C program as valid syntax...Maybe I'll ask for help from RC after lunch. Man, I'm dragging! Okay, well let's just try writing a frame to a file as binary. Phew -- how do I write binary to a file in C? Well I know that this example program already does that, writing each frame as "image" into the outfile</code> argument. So, tracing through the code</a> I find this call to fwrite</code></a>: const unsigned char buf = img->planes[plane]; const int stride = img->stride[plane]; const int w = vpx_img_plane_width(img, plane) * ((img->fmt & VPX_IMG_FMT_HIGHBITDEPTH) ? 2 : 1); const int h = vpx_img_plane_height(img, plane); int y; for (y = 0; y < h; ++y) { fwrite(buf, 1, w, file); buf += stride; } </code></pre> Omg, that's far too many parameters for a write function. What do all these mean? Ahhh, I'm in C, so man fwrite</code> (redacted below) is my friend: NAME</h3> fread, fwrite -- binary stream input/output </code></pre>SYNOPSIS</h3> #include <stdio.h> size_t fwrite(const void restrict ptr, size_t size, size_t nitems, FILE restrict stream); </code></pre>DESCRIPTION</h4> The function fwrite() writes nitems objects, each size bytes long, to the stream pointed to by stream, obtaining them from the location given by ptr. </code></pre>RETURN VALUES</h4> The function fread() does not distinguish between end-of-file and error; callers must use feof(3) and ferror(3) to determine which occurred. The function fwrite() returns a value less than nitems only if a write error has occurred. </code></pre></blockquote> Shockingly helpful, that. So going back to my fwrite</code> example: fwrite(buf, 1, w, file); </code></pre> buf</code> - this is a pointer that I'm going to copy from</li> 1</code> - is the size in bytes of each "item" I'm going to take take from buf</code></li> w</code> - is the number of "items" to read from buf</code></li> file</code> - is the destination where all those items are being written to</li> </ul> Not so scary now. Let's go back and try to write my frame out to a file: fwrite( frame, // (replaces buf) this is the raw frame data I want to store in a file 1, // (stays the same) frame_size, // (replaces w) this is the number of bytes in frame, all of which I want to write ); </code></pre> Okay, so I've now rewritten my inner loop to be a conditional (so that it only runs once) that looks like this now: if (vpx_video_reader_read_frame(reader)) { vpx_codec_iter_t iter = NULL; vpx_image_t img = NULL; size_t frame_size = 0; const unsigned char frame = vpx_video_reader_get_frame(reader, &frame_size); printf("\nframe_size=%zu\n", frame_size); fwrite(frame, 1, frame_size, outfile); } </code></pre> Compile and run against my IVF file and it appears to have worked! Here's a screenshot of the outfile inside Hex Fiend</a>. You can see that the number of bytes from my hacked-up C program and Hex Fiend are both 34974 bytes. Pair Programming Iterlude</h2> My fellow Recurser, Andy, offered to pair with me on figuring out all this and we were able to get a simplified C version of simple_decoder working: int main(int argc, char **argv) { int frame_cnt = 0; FILE infile = NULL; vpx_codec_ctx_t codec; const VpxInterface decoder = NULL; if (argc != 2) die("Invalid number of arguments."); if (!(infile = fopen(argv[1], "rb"))) die("fread: Failed to open %s for reading", argv[2]); decoder = get_vpx_decoder_by_fourcc(0x30385056); //info->codec_fourcc); if (!decoder) die("Unknown input codec."); if (vpx_codec_dec_init(&codec, decoder->codec_interface(), NULL, 0)) die_codec(&codec, "Failed to initialize decoder."); vpx_codec_iter_t iter = NULL; vpx_image_t img = NULL; size_t frame_size = 34974; const unsigned char frame = malloc(frame_size); if(fread(frame, 1, frame_size, infile) != frame_size) die("Nope"); if (vpx_codec_decode(&codec, frame, (unsigned int)frame_size, NULL, 0)) die_codec(&codec, "Failed to decode frame."); while ((img = vpx_codec_get_frame(&codec, &iter)) != NULL) { ++frame_cnt; } printf("Processed %d frames.\n", frame_cnt); if (vpx_codec_destroy(&codec)) die_codec(&codec, "Failed to destroy codec"); fclose(infile); return EXIT_SUCCESS; } </code></pre> This program basically just reads in the raw binary from a previous file (infile</code>) and has various configuration settings hardcoded for the decoder to be initialized and the frame size. It still doesn't work</h2> Unfortunately, the libvpx_native_sys</code> doesn't provide many of structs and things I need even to get this simplified simple_decoder to work.... I give up for today. Debugging Rust programs with lldb on MacOS 2018-01-10T00:00:00+00:00 I'll explain how to step through a Rust application using LLDB and illustrate some of the most basic commands. This post assumes MacOS throughout. Things are probably very different on Windows, but might be very similar on Linux and friends. Why debuggers in Rust</h2> There's no repl</a> for Rust, so its especially important to use other techniques to interact with Rust code; I believe that its unlikely there would ever be a Rust repl given the fact that Rust is a compiled language. Additionally, its very cumbersome to use the online playground</a> for understanding the code I'm actually working on right now. So, the main options for debugging code are (1) lean on the compiler and (2) lots of println!()</code>, which can only get you so far. Fortunately, Rust has C Application Binary Interface (ABI)</a> compatibility so we can mostly pretend that compiled Rust is C code. The Rust compiler team has also embedded debug symbols using the DWARF</a> protocol, so that means that we can use the always-fashionable GDB</a> and LLDB</a>. Some of the formatting isn't quite right, so Rust's package manager cargo</a> ships with a wrapper script for each: rust-gdb</code> and rust-lldb</code> that includes a python script for formatting purposes. 1 - Compile with debug symbols</h2> By default, cargo build</code> creates a "debug" build that contains the debug symbols. However, cargo install</code> does a "release" build with is optimized and contains no debug symbols. I haven't figured out to use LLDB with cargo install</code>, but it's probably very similar what I'm going in this post. $ cargo build -v Compiling mycrate v0.1.0 (file:///home/user/programming/mycrate) Running `rustc --crate-name mycrate src/main.rs --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=8b3e6af7e4113faf -C extra-filename=-8b3e6af7e4113faf --out-dir /home/user/programming/mycrate/target/debug/deps -L dependency=/home/user/programming/mycrate/target/debug/deps --extern log=/home/user/programming/mycrate/target/debug/deps/liblog-6d2adb16c0d397ce.rlib --extern mkv=/home/user/programming/mycrate/target/debug/deps/libmkv-391472e7b11b31e6.rlib --extern env_logger=/home/user/programming/mycrate/target/debug/deps/libenv_logger-8b11da9f68ecb368.rlib` Finished dev [unoptimized + debuginfo] target(s) in 13.4 secs </code></pre> Notice the Finished dev [unoptimized + debuginfo]...</code> on the last line. The "debuginfo" tells us we have produced a debugable binary. Digression - Compiling debug symbols using rustc directly</h3> If you don't use cargo</code> (I'm looking at you Dropbox!), the rustc</code> compiler supports two command line flags which will do this: -g</code></li> -C debuginfo=2</code></li> </ul> As far as I can tell, these appear to do exactly the same thing. There's no reason to pick one over the other. However, you may only use one or the other form but not both. Full example: rustc -g src/main.rs </code></pre>2 - Test loading your program in LLDB</h2> I got stuck here for quite a while, so I'm going make verifying the LLDB/Rust setup a whole step in this tutorial. Quick primer for folks new to LLDB/GDB like me -- these debuggers are invoked directly from the command line and passed an executable that they will target. The executable is not started -- by default there are no breakpoints set, so program would just run to completion without ever letting LLDB pause and explore. Okay, let's feed LLDB our program mainMYGITHASH</code> (which is in target/debug/deps/), by typing: $ sudo rust-lldb target/debug/deps/mainMYGITHASH </code></pre> Notice the path is target/debug/deps/mainMYGITHASH -- you must use the binary in the deps directory. See this cargo issue</a></li> On MacOS, sudo</code> seems to be required, otherwise I see errors about "codesigned" and nothing works. I kind of suspect Linux and friends are similar.</li> On my machine, lldb</code> is already installed though I have no idea if this was accidental or ships with MacOS by default. Linux systems may need install it, but I have no idea</li> Rustup</a> installs rust-lldb</code> in $HOME/.cargo/bin alongside rustc</code>, cargo</code>, and friends. Use rustup if you don't have rust-lldb</code>; then make sure that $HOME/.cargo/bin is in the PATH</code>.</li> </ul> You should see output like this: (lldb) command source -s 0 '/tmp/rust-lldb-commands.6F5RYs' Executing commands in '/tmp/rust-lldb-commands.6F5RYs'. (lldb) command script import "/Users/bryce/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/etc/lldb_rust_formatters.py" (lldb) type summary add --no-value --python-function lldb_rust_formatters.print_val -x "." --category Rust (lldb) type category enable Rust (lldb) target create "target/debug/deps/rcplayer-8b3e6af7e4113faf" Current executable set to 'target/debug/deps/rcplayer-8b3e6af7e4113faf' (x86_64). (lldb) </code></pre> If instead you see lots of warnings about missing symbols, instead go back and re-read this cargo issue</a>. TL;DR -- use the executable in target/debug/deps/ which ends in a git hash. Otherwise, at the time of writing this, its not guarranteed that LLDB will work. Exiting an LLDB session</h3> If LLDB is even close to working, the prompt will change to "lldb" as we saw in the expected output sample above. We're now inside a LLDB repl, and any time LLDB pauses or the exexecutable we're running finishes, we'll be dropped back into this repl. To get out of the repl, we have a couple of options: CTRL-D - works basically anywhere: ssh, bash, and most interactive command line tools</li> q - this works in LLDB, GDB (and also less</code>). Using q will cause LLDB to confirm with me that I really really want to quit</li> </ul> 2 - Load program with args</h2> Okay, so we know how to enter and exit a LLDB session with our Rust program at this point. Lots of the time, I want to feed arguments to my program while its being debugged or else nothing interesting will happen. LLDB makes this super simple. We just put our commands at the end of the rust-lldb invocation. Example: $ sudo rust-lldb target/debug/deps/mainMYGITHASH arg1 arg2 </code></pre> If that worked you should see a line in the terminal like this: (lldb) settings set -- target.run-args "arg1" "arg2 </code></pre>Change arguments between runs of your rust program</h3> At the (lldb)</code> prompt type this to change the argument to "foobar": settings set -- target.run-args "arg1" </code></pre>3 - Debug your rust program</h2> Set a breakpoint</h3> The b</code> command sets a new breakpoint taking the name of a function (or a regular expression) as an argument. (lldb) b my_fn_name </code></pre> If it worked, you'll see something like: Breakpoint 1: where = rcplayer`rcplayer::parse_element + 21 at main.rs:83, address = 0x00000001000113f5 </code></pre> See help b</code> for more details on specifying setting breakpoints in files at line numbers. I haven't quite figured this out yet in multi-file projects. To see the current list of breakpoints use breakpoint list</code> Run the program</h3> The r</code> command (short for run</code>). Example: (lldb) r </code></pre> The rust executable will run as a new process until lldb either (1) hits a breakpoint, at which point it will pause, OR (2) the process terminates. In either case, LLDB will present the repl. See variables in the current stack frame</h3> The frame variable</code> command prints out all the variables in the current stack. Example: (lldb) frame variable (mkv::elements::Element ) element = &0x100617190 (mkv::elements::Element ) element = &0x0 </code></pre>Print out a variable or expression</h3> The p</code> command prints out a variable (and sometimes expressions, though some trial error may be required). Example: (lldb) p element (mkv::elements::Element ) $0 = &0x0 (lldb) p element error: Couldn't apply expression side effects : Couldn't dematerialize a result variable: couldn't read its memory </code></pre>See some context from the source code of the program where execution is paused</h3> The list</code> command will show where are you now. If there's no process running, it seems to print out part of fn main(){}</code>'s code for me. Example: (lldb) list 87 content: Binary(ref bytes), 88 } => { 89 let raw_id = four_u8_to_u64(bytes.as_slice()); 90 let class = id_to_class(raw_id); 91 format!("SeekID ({:?})", &class) 92 } 93 Element { </code></pre>Summary</h2> To get ready to debug, make sure you have a debug build available. Run rust-lldb with sudo on the rust executable inside target/debug/deps/. The most important commands to debug a program are: b</code> - set breakpoint(s) before running the program</li> r</code> - start a new process</li> list</code> - shows the code where we paused</li> frame variable</code> - shows variables current state in the current frame</li> p</code> print an expression</li> CTRL-D - quits GDB immediately</li> </ul> Let me know about other tips you have on using LLDB in general or with Rust in particular! Join the conversation on Reddit</a>. Recurse Day 3: Port simple_decoder.c to Rust 2018-01-10T00:00:00+00:00 I want to learn more about how video codecs work so someday I can grow up to be a Jedi-master Video Engineer. To aid me on this epic quest, I'm doing the one week "mini-batch" at the Recurse Center</a>. Here's my notes from Day 3. Lesiurely morning</h2> I'm getting really tired from all the late nights, so I more or less took the morning to start some laundry (ut-oh...need to shuffle that a few hours ag...) and write a checkin on the Recurse Center chat program describing what I've done and plan to do today. Blogging about LLDB and Rust</h2> Then I immediately started doing other things! Specifically, writing this blog post about debugging Rust with LLDB on a MacOS</a>, but I think that was a great distillation of hard to find information into a succint and approachable format. I'll call that a success overall...just not my actual goal. Learning how to write a lisp</h2> Over falalel lunch, I had the best conversation about how one implements user-defined functions in a homemade lisp interpreter and then how to invoke that function later. Apparently, this requires a "special form" in the AST whenever the parser encounters the function keyword. This special form captures the function's: name</li> arguments</li> body as an AST</li> </ul> The whole thing is stored inside some kind of symbol table so that we can invoke it later. Okay, so we've parsed out this funciton into a usable form. Great! Now, let's evaluate our AST and handle a function invocation. (1) Lookup the function by name in our symbol table. (2) Bind the proper tokens to the function arguments. (3) Evaluate the function body's AST as normal. I think that mostly makes sense to me, though it'd take me a while to get that working. Port simple_decoder.c to Rust</h2> Okay, so I have this sample C program</a> doing exactly what I want (almost), and there's a generated crate for libvpx</a>, so I'm going projectto start by trying to compile the crate on my system... Make pkg-config happy</h3> No love, apparently I need to have a file name vpx.pc ("pc" means package config maybe?) to satisfy the needs of pkg-config. Its telling me to put that file into my PKG_CONFIG_PATH environment variable. Okay, let's go see if there's a vpx.pc somewhere in my libvpx build directory.... (Note to self: someday actually learn how to use find</code>. For now using exa --tree | grep .pc</code></a> to search recursively through the build dir. It's in the top level of build. Anyway, I found it -- time to look at PKG_CONFIG_PATH... env | grep PKG</code> gives me nothing, so...maybe I try export PKG_CONFIG_PATH=[my build dir]</code>. Hooray! Make Rust type checker happy</h3> That got to a whole new set of errors. All of these errors are about mismatched types: $ cargo build Compiling libvpx-native-sys v4.0.2 Compiling vp9_simple_decoder v0.1.0 error[E0308]: mismatched types --> src/main.rs:22:13 | 22 | ffi::VPX_CODEC_MEM_ERROR => Error::Mem, | ^^^^^^^^^^^^^^^^^^^^^^^^ expected u32, found enum `ffi::vpx_codec_err_t` | = note: expected type `u32` found type `ffi::vpx_codec_err_t` .... error: aborting due to 8 previous errors error: Could not compile `vp9_simple_decoder`. To learn more, run the command again with --verbose. </code></pre> Hmm, literally all of the errors are about u32</code> being expected by the function signature, but instead my code is giving back ffi::vpx_codec_err_t</code> typed data. The definition of vpx_codec_err_t</code></a> is: pub type vpx_codec_err_t = Enum_Unnamed3; </code></pre> ...and the definition of Enum_Unnamed3</code></a> (love that name by the way) is... pub type Enum_Unnamed3 = ::libc::c_uint; </code></pre> ...and the definition of c_uint</code> from libc</a> is...on my platform... type c_uint = u32; </code></pre> Not sure what I was expecting, but I assumed it would in fact be the type u32</code> the compiler seems to want. So, perhaps I could try to cast the type in my crate and keep going, OR I could try to understand why this would happen. First, I'll try to cast to just get this working and maybe later come back to learning why this type alias isn't working the way it maybe should. Wait wait wait -- I don't need to cast, I'll change the code I "borrowed" from the rustvpx companion crate</a> to expect ffi::vpx_codec_err_t</code> types instead of u32</code> since I know that these should always be the same on my platform. (What could go wrong?!). That worked! Make ld happy</h3> New errors now: $ cargo build Compiling vp9_simple_decoder v0.1.0 error: linking with `cc` failed: exit code: 1 | = note: "cc" "-m64" "-L" "/home/user/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/home/user/vp9_simple_decoder/target/debug/deps/vp9_simple_decoder-761a7810146c7575.vp9_simple_decoder0.rust-cgu.o" "/home/user/vp9_simple_decoder/target/debug/deps/vp9_simple_decoder-761a7810146c7575.vp9_simple_decoder1.rust-cgu.o" "/home/user/vp9_simple_decoder/target/debug/deps/vp9_simple_decoder-761a7810146c7575.vp9_simple_decoder2.rust-cgu.o" "/home/user/vp9_simple_decoder/target/debug/deps/vp9_simple_decoder-761a7810146c7575.vp9_simple_decoder3.rust-cgu.o" "-o" "/home/user/vp9_simple_decoder/target/debug/deps/vp9_simple_decoder-761a7810146c7575" "/home/user/vp9_simple_decoder/target/debug/deps/vp9_simple_decoder-761a7810146c7575.crate.allocator.rust-cgu.o" "-Wl,-dead_strip" "-nodefaultlibs" "-L" "/home/user/vp9_simple_decoder/target/debug/deps" "-L" "/usr/local/lib" "-L" "/home/user/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/home/user/vp9_simple_decoder/target/debug/deps/libvpx_sys-3b31031fb7744e6c.rlib" "/home/user/vp9_simple_decoder/target/debug/deps/liblibc-731e9e00d9493fb1.rlib" "/home/user/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libstd-3bdc66d380ab3ae0.rlib" "/home/user/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liballoc_jemalloc-7d37db020b297d4d.rlib" "/home/user/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liballoc_system-6020283742edae37.rlib" "/home/user/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librand-a9c96b55be5fbf12.rlib" "/home/user/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-cea338bc19530d4f.rlib" "/home/user/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libunwind-ec9c6c2125b5c0ae.rlib" "/home/user/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liblibc-b32efaa792018eef.rlib" "/home/user/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liballoc-010885f4927e31a8.rlib" "/home/user/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libstd_unicode-d8d07f9be800bfd6.rlib" "/home/user/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcore-f4176b2f23d80db2.rlib" "/home/user/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-6e4af26e08893557.rlib" "-l" "vpx" "-l" "m" "-l" "System" "-l" "resolv" "-l" "pthread" "-l" "c" "-l" "m" = note: ld: library not found for -lvpx clang: error: linker command failed with exit code 1 (use -v to see invocation) error: aborting due to previous error error: Could not compile `vp9_simple_decoder`. To learn more, run the command again with --verbose. </code></pre> Looks like the system linker ld</code> can't find the library vpx ("ld: library not found for -lvpx"). That's probably a think I can duckduckgo.... The first few results are not super helpful. Okay, let's actually try to grok that command cargo spat out: cc</code> - I know this is a c compiler (no idea how I know this), probably an apple version of gcc..yep.</li> -L "/path/to/rust/library"</code> - looks like we're telling cc</code> about other libraries to link against by providing the full path. That is most of the monstruously large error message above.</li> -l vpx</code> - I'd venture a guess this is a library but its one that the linker or compiler is supposed to just know where it is.</li> </ul> Okay, lets figure out how cc</code> uses the -l</code> flag for sure...nope, there's no entries for this in man cc</code>. Apparently, this is undocumented feature. Okay, well cargo error says something about ld</code> so lets try man ld</code> and looking for something about where it looks for libraries (my hypothesis for what -l vpx</code> means). Sure ensure, there's a description of how this works: $ man ld ... Search paths ld maintains a list of directories to search for a library or framework to use. The default library search path is /usr/lib then /usr/local/lib. The -L option will add a new library search path. The default framework search path is /Library/Frameworks then /System/Library/Frameworks. (Note: previously, /Network/Library/Frameworks was at the end of the default path. If you need that functionality, you need to explicitly add -F/Network/Library/Frameworks). The -F option will add a new framework search path. The -Z option will remove the standard search paths. The -syslibroot option will prepend a prefix to all search paths. </code></pre> So, if I know which lib is the library file for vpx, I can stick that file in /usr/lib or one of the other directories mentioned. Okay, wow I hate not knowing how C works at all. Phew...the rust libraries mostly have an rlib extension. Maybe there's an rlib in the libvpx/build directory? No, not even one file. Okay, maybe "dylib"? Nope. What about just things that have "lib"? Yep -- there's a bunch .a files, like libvpx.a. Hmm, that sounds plausible. What does .a even mean? I know .o is for object files produced during a C compilation, whatever those are... Okay, well some guy named Mike has this on his blog about .a files</a> and compiling C: There are other kinds of files as well, notably libraries (".a" files) and shared libraries (".so" files), but you won't normally need to deal with them directly. </blockquote> Oh, yeah! I've totally worked with .so files on linux before. A file ending in ".a" on MacOS is like a ".dll" for Windows or a ".so" for Linux. TIL! Not pwning my own system directories</h3> Okay, let's copy that libvpx.a into /usr/lib and see what happens...Uh, apparently thats not okay with Apple: cp: /usr/lib/libvpx.a: Operation not permitted </code></pre> There's a lot of symlinks in there, maybe I can just symlink libvpx.a? Nope, same result. I really feel like I should be able to do this, so there's something specific about that directory I don't understand. Sure enough, there's a great explanation of how Apple has deliberately locked down that directory</a> to keep us all safe. There's also an alternative here. Use the DYLD_LIBRARY_PATH</code> environment variable instead. Let's give that a whirl. No difference whatsoever. Same result we started with. Hmm, maybe let's try a different approach. Maybe I could get more info out of cargo by using the verbose flag. Let's try: $ cargo build -v Fresh libc v0.2.35 Fresh pkg-config v0.3.9 Fresh semver-parser v0.7.0 Fresh libvpx-native-sys v4.0.2 Compiling vp9_simple_decoder v0.1.0 (file:///Users/bryce/programming/vp9_simple_decoder) Running `rustc ...` error: linking with `cc` failed: exit code: 1 | ... = note: ld: library not found for -lvpx clang: error: linker command failed with exit code 1 (use -v to see invocation) error: aborting due to previous error error: Could not compile `vp9_simple_decoder`. Caused by: process didn't exit successfully: `rustc --crate-name vp9_simple_decoder src/main.rs --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=761a7810146c7575 -C extra-filename=-761a7810146c7575 --out-dir /Users/bryce/programming/vp9_simple_decoder/target/debug/deps -L dependency=/Users/bryce/programming/vp9_simple_decoder/target/debug/deps --extern vpx_sys=/Users/bryce/programming/vp9_simple_decoder/target/debug/deps/libvpx_sys-3b31031fb7744e6c.rlib --extern libc=/Users/bryce/programming/vp9_simple_decoder/target/debug/deps/liblibc-731e9e00d9493fb1.rlib -L native=/usr/local/lib` (exit code: 101) </code></pre> Okay, now that rustc</code> invocation at the bottom also has -L native=/usr/local/lib</code> in it....I can definitely put anything in that directory I want to. Let's try it! $ ln -s /home/user/libvpx/build/libvpx.a /usr/local/lib/libvpx.a $ cargo build Compiling vp9_simple_decoder v0.1.0 (file:///Users/bryce/programming/vp9_simple_decoder) Finished dev [unoptimized + debuginfo] target(s) in 0.66 secs </code></pre> Woohoo! I compiled the thing. Oh man that's exciting. Time for pizza and human contact...I'll see what Andy's up to with his Rust lisp interpreter. Porting a few lines of simple_decoder</h3> Okay, I've tried to do a mechanical port of simple_decoder.c file directly using the low-level C bindings in crate libvpx-native-sys</a>. However, the first interesting thing that happens in simple_decoder, is that we open a file and try to parse it using vpx_video_reader_open()</code>...which isn't included in the crate I'm using. There's some mention of bindgen</a> in some of `the code, so I go and look at the infamous bindgen crate</a> and its very accessible tutorial</a>. Step 1 in the tutorial is: Add bindgen as a Build Dependency Declare a build-time dependency on bindgen by adding it to the [build-dependencies] section of our crate's Cargo.toml metadata file: [build-dependencies] bindgen = "0.26.3" </code></pre></blockquote> ...Uh oh, there's no bindgen</code> dependency inside Cargo.toml</a> in libvpx_sys crate...instead there's a PNaCl helper create. What the flip is PNaCl and why does it sound so familiar? A few duckduckgo's later, I find the project description</a>: Native Client (NaCl)</h3> Native Client enables the execution of native code securely inside web applications through the use of advanced Software Fault Isolation (SFI) techniques. Native Client allows you to harness a client machine’s computational power to a fuller extent than traditional web technologies. It does this by running compiled C and C++ code at near-native speeds, and exposing a CPU’s full capabilities, including SIMD vectors and multiple-core processing with shared memory. While Native Client provides operating system independence, it requires you to generate architecture-specific executables (nexe) for each hardware platform. This is neither portable nor convenient, making it ill-suited for the open web. Portable Native Client (PNaCl)</h2> PNaCl solves the portability problem by splitting the compilation process into two parts: compiling the source code to a bitcode executable (pexe), and</li> translating the bitcode to a host-specific executable as soon as the module loads in the browser but before any code execution.</li> </ul> </blockquote> Right, so like WebAssembly but that only works on Chrome and friends. Oh, and then I see the WebAssembly migration guide</a>: Given the momentum of cross-browser WebAssembly support, we plan to focus our native code efforts on WebAssembly going forward and plan to remove support for PNaCl in Q1 2018 (except for Chrome Apps). </blockquote> Okay, why is this crate using PNaCl? I'm so confused. At any rate, I have to stop for the day and go to sleep. Open Questions</h2> Should I contribute patches on top of rustvpx crate to get the bits I need to decode? Or should I rewrite the example C program to only use functions that are available in libvpx-sys crate?</li> Do any of the functions inside libvpx actually work at all?</li> Could I get Doxygen to produce the documentation for libvpx from the C code? Or is there a copy of the docs floating around on the interwebs?</li> Should I go back and just work on nom / mkv.rs? I might ping that maintainer again and call it a night. We'll see what he says in the morning</li> </ul> Recurse Center Day 2: Seeks and Blocks 2018-01-09T00:00:00+00:00 I want to learn more about how codecs work so that I one day I can grow up to be a Jedi-master Video Engineer. Here's my notes from Day 2. Step 0 - Actually Reading the Specs</h2> The Matroska specs helpfully include a diagram</a> which I've ignored up until now which answers about 95% of my questions from last night. To quote the specs: The Track section has basic information about each of the tracks. For instance, is it a video, audio or subtitle track? What resolution is the video? What sample rate is the audio? The Track section also says what codec to use to view the track, and has the codec's private data for the track. The Clusters section has all of the Clusters. These contain all of the video frames and audio for each track. </blockquote> There's a lot more I learned today, but I'm going to focus on doing it and possibly write down interesting things later. Recurse Center Day 1: VP9 != Matroska 2018-01-08T00:00:00+00:00 I want to learn more about how video codecs work so someday I can grow up to be a Jedi-master Video Engineer. To aid me on this epic quest, I'm doing the one week "mini-batch" at the Recurse Center</a>. Here's my notes from Day 1. Step 1 - Find a VP9 sample video</h1> Tears of Steel</a> is legendary for all the different kinds of lighting, motion, color, and uh...things one might need to compress with a video encoder. I've been wanting to play with that video for a long time. It's licensed as creative commons and made by the good people at Blender, and its actually pretty fun to watch. A quick search on duckduckgo shows you can download a webm version of the file. This is great news because webm videos use the vp9 codec, which is what I'm trying to use my weeklong training program at the Recurse Center in NYC to learn about. Except that webm doesn't only mean vp9...it also means vp8. Time to transcode the video using ffmpeg to use vp9: ffmpeg -i tears_of_steel_1080p.webm -c:v libvpx-vp9 -c:a libopus output.webm </code></pre> -i</code> means input file</li> -c:v</code> specifies the video codec for all streams in the file, and lipvpx-vp9</code> means use the libvpx library's vp9 codec</li> -c:a</code> is the same thing, but for audio</li> output.webm</code> is the output file that ffmpeg will create. It infers lots of things from the fact I've used the .webm file extension</li> </ul> Oh man...ffmpeg is going so slow. The output looks like this: frame= 7540 fps=3.6 q=0.0 size= 13211kB time=00:05:14.31 bitrate= 344.3kbits/s speed=0.15x </code></pre> The video is 12 minutes and 14 seconds long, but I've already waited for at least an hour... Two hours later...</h2> I've retranscoded to vp9! I've managed to fix three bugs in my blog while waiting for the transcode to finish. If I had to guess, the job took so long because I didn't specify any quantization parameters (that's the "q=0.0")...I really need to read up on exactly what quanta are on wikipedia</a>. A quick digression into quantization...with a picture!</h3> The intro to that article was really helpful. Apparently, quantization basically just means taking some information that has an infinite range of possible values and chopping it into a finite number of discrete values. Actually my picture used in the header of this blog is a great example of quantization. The original photo on the left has tons and tons of colors, but I've forced the header image (on the right) to use fewer than 10 colors. Quantization. Not so scary a concept from 20,000 feet. Using mkvinfo</code></a>, I can see that my output.webm is actually VP9 now. Phew! Okay, I have a sample video. Before coming to the Recurse Center, I'd been reading up on the Matroska format</a> to get ready to build my decoder...but total fail, that's the just the container format and contains 0% of video information. Now the awesome rust libraries I found are basically worthless...Hmm, time to rethink the project? Nope, time for boba tea and a little reflection. Boba Interlude</h2> It's raining and Boba was delicious. I'm just starting to skim the table of contents on the VP9 spec</a>, and it's a little overwhelming. I've asked for help for the most excellent video-dev slack org</a> about suitable alternative projects. Here's some ideas that have come to mind: Rewrite a simplified ffmpeg-like-tool that consumes libav or libvpx to transcode or decode a video</li> Just read through the whole VP9 spec</li> Contribute to FFMPEG/VP9/AV1 open source projects</li> </ul> Might be helpful to take a step back here. When I leave Recurse Center, I want to understand more about how to efficiently encode and troubleshoot videos especially those using VP9. So, there's actually a lot of value in just understanding the webm / Matroska format...There's also value in understanding how a video gets encoded using VP9 to understand the quality/filesize tradeoffs and what encoding parameters can be tuned a bit better. Actually knowing the super deep technical areas of VP9 might not be totally necessary... I just got a great suggestion from Slack -- try to contribute to the AOM Analyzer</a> project. I remember seeing this before -- and its really cool. However, it's really broken on Firefox (my browser of choice since 2014) and Chrome. Maybe another time I'd look at that project again, but not today. Contributing to ffmpeg or AV1 would be super amazing, but I think I need a stronger foundation in how the codecs themselves work. FFMPEG is notoriously hard to contribute to, but I wonder if there are beginner issues for AV1...ah, nope there's only an issue tracker</a> for AV1 and no obvious on-ramp there, although there is documentation about building and testing the project. Still seems a little too pie-in-sky. Decision Time</h3> Why don't I use the Matroska rust library to extract the VP9 bits of a video and try decoding the first frame first using libvpx; once I've got that working, I'll try to read the spec to do that myself...Great! This could keep me busy for at least the whole week, but I'd likely learn a lot about how the codec works and build a thing that does something with a quick win and then lets me go deeper. The Matroska crate</a> I've been playing with doesn't provide any access to the raw bytes, so there's no way that will work. Maybe, I could just contribute to that project</a> a way to expose the lower level parsing logic from that crate. Wait...I should really look at that other library mkv</a>. Yeah! THis does exactly what I want! It spits out typed events corresponding to the start and end of each EBML element as it does a single pass through the file. I'll build on top of that then! Also, the Matroska stuff does actually seem to be super valuable. I really need to have a handle on the container format in order to find the image data for vp9 to process, and the webm project requires a lot of knowledge of the Matroska format...so it doesn't look like a waste of time after all. Playing with libvpx</h2> Okay, I'll just start by making a new GitLab project</a> and pushing the little bit of code I have now there...and I'll add a README with a link to Tears of Steel source I'm working with (I would love to get a "Bars and Tones" sample video for easier sample...). A bowl of Ramen and a karafe of sake later, it looks like the mkv crate</a> is exactly what I want for parsing Matroska packaging. It gives me the actual binary, the offset, and an enum variant for each Matroska ID. Amazing! So, now I need to see if I can interface this with libvpx... 9:45pm -- time to head back to my apartment. Okay, I've compiled lipvpx</a> by hand and looked closely at the most simple decoder example</a>. I've also quickly skimmed the vpx crate</a> and it looks like all the necessary C functions have rust bindings</a>. Unfortunately, none of my webm files work when I run them through the example program from libvpx. Running them through mkvalidator gives me errors on all of them, but VLC has no trouble with any of these files and ffprobe also is very happy with them. Oops! Stupid user error on my part. Looks like that script only works on IVF files -- Today I Learned IVF means vp8. I swear I have never heard of IVF before today, and I assume it's not something I'll hear a lot about going forward. So, that script appears to have succeeded in creating a massive pile of images, but I'd love to verify. Time to download ffplay</code> so I can follow the script that simple_decoder.c told me to run: ffplay -f rawvideo -pix_fmt yuv420p -s 512x288 /tmp/out9-ivf </code></pre> Oh man...I have ffprobe and ffmpeg, but none of the other ff tools. I can never remember the write way to install ffmpeg and get ALL the tools, so here's how: Go to https://ffmpeg.zeranoe.com/builds/</li> Click the blue Download button</li> Copy everything from the bin/ directory of the unpacked tarball into /usr/local/bin/</li> Make sure it worked which ffplay</code></li> </ol> Let's try that ffplay</code> command again now... Success! Open Questions</h2> How does VP9 fit into webm? The code sample for simple_decoder.c makes it look like libvpx is actually parsing some matroska...but running mkvalidator seems to disprove that theory. No EBML found in the file.</li> What's the difference between a Seek</code> element and a CuePoint</code>?</li> Where does encoded audio data actually go inside a Webm file? Inside Blocks</code> / SimpleBlocks</code>?</li> Why is timing so complex inside webm?</li> How can I feed raw bytes into libvpx for vp9? Hmm, actually I know a vp9 codec core contributor...I should ask him about this!</li> What can I reasonably try to accomplish tomorrow? I really think unpacking SeekPositions</code> and possibly the Video</code> element from the Webm file will point me to the correct places inside the file to find data to feed the vp9 codec. Maybe I'll just try to wrap my head around those elements better.</li> </ul> Strategies for Returning References in Rust 2017-03-04T00:00:00+00:00 I'm going to demonstrate the situation where I most often have friction with The Borrow Checker and provide a few patterns I use to make my code compile. Let's pretend we're trying to connect to an imaginary Postgres database server using a connection pool. The API for this imaginary library requires us to first initialize a connection Pool</code> with the connection string. Once the pool is initialized, we call connect()</code> on it to get access to a usable, owned Connection</code> for queries. Why Returning References Can Be Difficult</h2> Let's start with a naive approach that won't even compile: // This won't compile! fn connect(url: &str) -> &Connection { let pool: Pool = postgres::connection_pool::new(url); let connection: Connection = pool.connect(); connection.set_database("startup.io"); &connection } </code></pre> The compiler will complain about the lifetime of &Connection</code> not lasting long enough. Why? The compiler wants to deallocate the Connection</code> object at the end of the function, but it knows that the &Connection</code> reference we're trying to return points to that struct. The Book does a great job of explaining owned values</a>, borrowing</a>, and lifetimes</a>; but I'll provide my own version here. I'm doing a lot of generalizing and glossing over complexities in the next few paragraphs. If you understand why my sample code won't compile, feel free to skip ahead. A Digression into Memory Allocation</h3> A computer has a fixed amount of physical RAM which we typically refer to as "memory." As we write programs, all the data we process is represented in memory at some point during the execution of the program. We call the process of assigning memory to a specific running process "allocating" memory, and releasing ownership of memory "deallocating." In the simplest case, the newly allocated memory stores the data we assign to the variable. For other kinds of data, we store the memory address of another variable. We call this second kind of variable a "pointer" because it "points" to other memory. (I'll also use the term "reference" interchangeably with "pointer.") Pointers exist mostly to (1) avoid duplicating data in memory and (2) provide a predictably sized variable which represents unpredictably sized data (such as user input or a data from a network call). A Brief Digression on Garbage Collectors</h4> Many languages including Java and Python use a "garbage collector" to decide when a process should free memory. Every so often, the garbage collector will (1) interrupt whatever the programmer's code is doing, (2) hunt for memory that doesn't have pointers to it, (3) and free that memory. Garbage collectors operate at "runtime," while the program is running. This means that programmers can let the garbage collector figure out when to free memory automatically, which makes garbage collected languages easier and safer to write, but sometimes slower and with unpredictable memory usage. Some languages don't use a garbage collector. In C, the programmer must decide when to free up memory manually, which is difficult to do correctly. If the programmer tries to use a pointer after the memory has been freed, Bad Things can happen -- things like crashing the program or allowing a hacker to gain root access. What makes Rust unique is that although it doesn't use a garbage collector, it still guarrantees that pointers will be safe to use. How Rust Achieves Memory Safety without a Garbage Collector</h4> The Rust compiler uses the metaphor of home "ownership." If memory is a house, then exactly one variable is the owner of that house and the death of that owner triggers an estate sale (aka deallocation). A variable owns memory when the programmer assigns data to it. When the owner goes out of scope and dies, that memory is deallocated. Because the compiler always knows when a variable goes out of scope, it always knows when to deallocate memory at compile time so Rust programs don't need to pause and garbage collect while running. As in real estate, ownership can be transferred. Like selling a house, assigning one variable to another variable transers ownership. We say that the value has been "moved." Similar to how selling my home means moving my stuff, moves in Rust copy data to a new place in memory. This is called "move semantics." You might think that if a large amount of data is moved then we're probably wasting RAM since the data must exist in at least two locations in memory. However, the Rust compiler heavily optimizes code when using the --release</code> flag, so most of the time the compiler can see that we're about to waste memory and simply reuse the existing memory location instead of stupidly copying. Another problem introduced by move semantics is that you often want many variables to be able to access the same data from different places in your program. Ownership would seem to make that impossible since only one variable can own a memory location at a time. However, Rust also allows "borrowing" of memory. The programmer borrows memory by creating a pointer to it. If memory is a house and one variable is the owner, pointers are the renters who temporarily use that memory. If the owner goes out of scope and dies, pointers can no longer legally use that memory. The Rust compiler acts like a property management agency by tracking all ownership lifetimes and borrowing to make sure that no memory is being squatted on illegally. This aspect of the Rust compiler is known as The Borrow Checker, and it's the thing which makes Rust uniquely Rust. Back to the code sample</h3> // This won't compile! fn connect(url: &str) -> &Connection { let pool: Pool = postgres::connection_pool::new(url); let connection: Connection = pool.connect(); connection.set_database("startup.io"); &connection } </code></pre> Going back to our example, there's only one scope involved: the entire body of fn connect() {..}</code>. So, at the end of connect()</code>, all variables declared inside the function will be deallocated. Invoking pool.connect()</code> creates an unnamed Connection</code> object in this scope, but because that object is created inside of the connect()</code> scope, it also must be deallocated at the end of function. This means that after we return the pointer to the Connection</code>, there won't be a Connection</code> anymore! This is what the compiler is trying to tell us: that pointer will refer to freed memory after the end of the function. Patterns for Returning References</h2> Pattern 1: Return Owned Values</h3> Give up on references, and just return a full copy of the value. Depending on the use case, this might be a good solution, and it's often the easiest way to appease The Borrow Checker. fn connect(url: &str) -> Connection { let pool: Pool = postgres::connection_pool::new(url); let connection: Connection = pool.connect(); connection.set_database("startup.io"); connection } </code></pre> There are two changes needed in this trivial case: Remove the &..</code> in the return value and use ..</code> instead</li> Remove the &</code> in the function body</li> </ul> Some types, like str</code> and Path</code> are only intended to be used with references, and they have a sibling type which is intended to be used only as an owned value. For these types, if we just try removing the &</code> from our code, we'll still get compile errors. Here's an example that won't compile: // This won't compile! fn returns_a_path() -> Path { let path: Path = somehow_makes_path(); path } </code></pre> For these types which can only be used as references, look for an impl</code> of the trait ToOwned</code>. How it works is ToOwned</code> consumes a shared reference and copies the value into a new owned reference. Here's an example of leveraging the ToOwned</code> trait: fn returns_a_pathbuf() -> PathBuf { let path: &Path = somehow_makes_path(); let pathbuf: PathBuf = path.to_owned(); pathbuf } </code></pre> Pros: Low Effort - generally, converting a return value from a shared reference to an owned value is really easy. I can usually get something to compile quickly and refactor later using this technique.</li> Widely Applicable - almost anywhere we can return a reference, we could also return an owned copy of that value</li> Safe - a new copy of the value cannot corrupt memory elsewhere, even when moving the data between threads</li> </ul> Cons: Synchronization - this value won't be changed if we change the original value.</li> Memory - we'll possibly be wasting memory by making identical copies of some piece of data (usually won't happen)</li> </ul> To me, using this technique often feels like a kludge because I've spoken to the Borrow Checker instead of expressing the core logic of my application. Think hard about how many places need to mutate the value in question or if they can all simply read the same value this way. If this is an unchanging value, returning an owned value can be a great solution. Pattern 2: Return Boxed Values</h3> Let's move the database connection off the function stack and into the heap. In our own types, we'll need to explicitly heap allocate data using the Box</code> struct from the standard library. So, refactoring our above code to use the heap would look like this: fn connect(url: &str) -> Box<Connection> { let pool: Pool = postgres::connection_pool::new(url); let connection: Connection = pool.connect(); connection.set_database("startup.io"); Box::new(connection) } </code></pre> There are two changes: The function signature now has Box<..></code> as the return type instead of &..</code></li> The function body instantiates a new box wrapping the connection expression Box::new(..)</code> instead of &..</code></li> </ul> The boxed value is an owned struct (not a reference) so it can be returned from the function without angering the borrow checker. When the box itself goes out of scope, the heap allocated memory will be freed. Pros: Memory - The only additional memory (beyond the boxed value) is the pointer to the heap, and pointers are only a few bytes</li> Applicability - Almost any code which uses std</code> can leverage this technique</li> </ul> Cons: Indirection - we'll need to write more code in our type annotations, and we may need to understand how to leverage the Deref</code> trait to work with boxed values</li> Overhead - It's more complicated to allocate memory on the heap and this may incur a runtime penalty.</li> </ul> For types like usize</code>, bool</code>, f32</code>, and other primitive types, it can be a code smell if I find myself boxing these values. Instead I normally return a copy of these types instead. For dynamically growable types like Vec</code>, String</code>, and HashMap</code>, these types already use the heap internally so you're not necessarily gaining much by boxing them. If performance or memory usage matter, profile your own code under realistic conditions to determine if boxing your return value improves or degrades performance relative to another pattern. As with the previous pattern, you want to thing about mutability -- how many places am I reading this value? How many of those places am I going to be writing to this value as well? Boxing in conjunction with Atomic Reference Counting (Arc</code> type) can make mutably sharing a value across threads possible if you need to. If you don't need that, boxing might just be slowing down your program or wasting memory. Pattern 3: Move Owned Values to a Higher Scope</h3> This technique involves reorganizing code to help us leverage references passed into functions. Here's an example: fn setup_connection(connection: &Connection) -> &Connection { connection.set_database("startup.io"); connection } fn main() { let pool = postgres::connection_pool::new(url); for _ in 0..10 { let connection = setup_connection(&pool.connect()); // Do something with connection here } } </code></pre> We achieved this by: moving both the Pool</code> and Connection</code> objects to a scope outside the function</li> changing the function signature to take in a reference to the connection we wanted to modify</li> returning a reference to the same memory as the memory borrowed in the argument</li> </ul> At the end of setup_connection()</code>, no new structs were declared inside of it. This means nothing needs to be deallocated. Because connection</code> was declared inside the for</code> loop, it lives for one iteration of the loop. In the next iteration, a new connection is allocated. Basically, we can only get the lifetimes out of a function that we put into the function. In this particular case, no matter how many times we call setup_connection()</code>, only one Pool</code> object will be allocated, compared to having many pools allocated in the other patterns. Pros: Memory - this pattern avoids heap allocating and writing boilerplate code, so it's the memory efficient and elegant</li> Aroma - this pattern is often the natural result of good code organization that reduces unnecessary work; its like a good code "aroma"</li> </ul> Cons: Complexity - this pattern requires deep understanding of the application and data flows, and usually means rewriting several different areas of the code</li> Applicability - many times this pattern can't be used</li> Rigidness - using this pattern might make refactoring the code more difficult</li> </ul> This pattern is likely to work when we have an owned value which "emits" other borrowed objects. If we can move the owned struct to a higher scope, we can pass referencecs to helper functions. When I'm looking to refactor and/or reduce memory usage, I examine if this pattern might be an option. Pattern 4: Use Callbacks not Return Values</h3> This technique was suggested by mmstick on reddit</a>. If you've ever written JavaScript with callbacks or used dependency injection in Java, you'll feel right at home here. The basic idea is to avoid fighting the Borrow Checker by passing a closure into the function. Example time: fn connect_and_attempt<F>(pool: &Pool, action: F) -> Option<String> where F: Fn(Connection) -> Option<String> { let connection: Connection = pool.connect(); connection.set_database("startup.io"); action(connection) } fn main() { let pool: Pool = postgres::connection_pool::new(url); let result = connect(&pool, |connection| { // do something with connection and return an option Some(output) }); } </code></pre> The key difference over pattern 3 is that we pass an anonymous function |connection| { .. }</code> into connect_and_attempt</code> and we never return the connection</code> object at all. This means we didn't have to fight the Borrow Checker. Pros: Elegant - avoid fighting the borrow checker</li> Decoupled - helps isolate application logic from I/O or dependencies</li> </ul> Cons: Inflexible - closures in Rust are more complicated than JavaScript and might require boxing in certain situations.</li> Complexity - requires a deeper knowledge of Rust and mastery of closures.</li> </ul> As noted by mmstick, this pattern can make unit testing really easy because we can decouple blocks of code and test their logic without setting up the full environment. We can pass dummy callbacks into the system under test. Additional Resources</h2> Reddit Comments</a> -- I'm tremedously greateful to all of the many people who chimed in on Reddit about this article -- I learned so much!</li> Let's Clone a Cow - New Rustacean Podcast</a> - Chris Krycho explains how to do some sophiscated things with memory management in Rust while satisfying the Borrow Checker</li> </ul> More patterns?</h2> Please, if you have experience using more patterns than I've enumerated here, please share them! I'd love to incorporate your patterns into this article as a sort of reference of techniques for newbies on the Borrow Checker. Faster CI Debugging with GitlabCI 2017-02-24T00:00:00+00:00 GitlabCI</a> can help you debug many problems that happen in cloud based continuous integration by allowing you to run the build locally. If you're leveraging docker to run your builds, it can also eliminate the "It works on my machine -- but not in CI" class of problems as well. I'm going to walk you through setting up gitlab-runner</code> locally, and show you how to start using it to speed up your debugging process. I'll explain how the architecture of GitlabCI allows us to do build locally, and some gotchas you may encounter. Assumptions</h2> The main assumption is that you're interested in or already using Gitlab. Not convinced you should switch? Read why I use Gitlab</a>. You Already Have GitlabCI Setup for Your Project</h3> This post isn't very helpful unless you've already setup GitlabCI using a .gitlab-ci.yml file. If you haven't already done that, checkout the getting started guide</a>. You Should Install Docker</h3> If your builds can run on Linux, you really should be using the Docker executor. Even if your machine runs on Windows, MacOS, or something else that supports Docker, you should install Docker. The main benefits that accrue to using Docker with GitlabCI are: Docker basically guarrantees that testing on your machine will produce identical results with a remote CI server (which is kinda the whole point of doing this). Without docker, you're much less likely to discover why something worked locally but broke in CI.</li> Docker can ensure that the testing environment configured to our exact specifications. The $PATH</code> can be configured ahead of time, various binaries and utilities installed, DNS configured, security restrictions imposed, etc, etc. The tradeoff is that docker will use more memory and be slower running on bare metal. And docker pull... why can't it be faster???</li> </ul> The main reason you might not care about installing Docker is if you want to build and test on a platform that isn't Linux. Docker can entirely capture all aspects of a local Linux system, but it can't capture Windows, Mac, Solaris, or BSD systems (to name a few). If you specifically want to reproduce issues occurring on those platforms, Docker won't help you. You'll need to find some other way to make sure your local machine is an exact replica of the remote CI server so that you can locally debug those issues. So, if you haven't already installed Docker, here's how to do it in Linux: $ curl -sSL https://get.docker.com/ | sh </code></pre> For MacOS or Windows, go to the Docker download page</a>, download your installer, and run it. How Does Gitlab Runner Relate to GitlabCI?</h2> GitlabCI is composed of a "dashboard" and many "runners". The GitlabCI Dashboard</h3> The dashboard is part of Gitlab and can be controlled by: doing a git push to your gitlab instance</li> starting, cancelling, or retrying a pipeline in the web interface</li> making API requests to the gitlab instance</li> </ul> The dashboard is the command and control center. It doesn't do the work; it just delegates and supervises a pool of runners. When the dashboard receives instructions to start a pipeline, it adds that pipeline to a queue, and then feeds work from the queue to available runners. The Gitlab Runners</h3> The runners are individual computers that are running the Gitlab Runner binary. Typically, runners register themselves with the dashboard using a secure token generated by the dashboard. Runners can have names and tags; they receive instructions from the dashboard, execute the command and continuously report back results of running pipelines and jobs. It's very important that registered runners have as close to 100% network connectivity as possible to ensure that jobs can be run on demand. Typically, the runners are installed as a system service or daemonized. Runners can also operate without connecting to a Gitlab instance. In this offline mode, they have limited functionality, but they can read a local git repository and directly execute jobs listed in a .gitlab-ci.yml. Offline runners must be invoked directly on the command line to execute; they don't run daemonized in this mode. In this guide, we'll be installing Gitlab Runner on your local machine and running in this offline mode to debug CI problems locally. Install Gitlab Runner on MacOS</h2> At time of writing these are the official steps: $ sudo curl --output /usr/local/bin/gitlab-ci-multi-runner https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-ci-multi-runner-darwin-amd64 $ sudo chmod +x /usr/local/bin/gitlab-ci-multi-runner </code></pre> NOTE: For the most current instructions, check the official documentation</a>. Unfortunately, Gitlab Runner has different names on MacOS and Linux. I tend to prefer the shorter name, so I'd recommend you add a symlink in your path to the Linux name (gitlab-runner</code>). I'll be referring to the command as gitlab-runner</code> in the rest of this post, so if you don't symlink as recommended below, use gitlab-ci-multi-runner</code> in your terminal instead of gitlab-runner</code>. $ sudo ln -s /usr/local/bin/gitlab-ci-multi-runner /usr/local/bin/gitlab-runner $ which gitlab-runner /usr/local/bin/gitlab-runner </code></pre>Install Gitlab Runner on Ubuntu</h2> For Ubuntu (and presumably Debian as well), you'll want to use Gitlab's PPA so that you're always on a current version of GitLab Runner. At the time of writing, you can add the Gitlab PPA like so: $ curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh | sudo bash </code></pre> (Note that this may change over time. Here's the current installation instructions</a> for Gitlab Runner.) Let's install gitlab-runner</code> itself now: $ sudo apt-get install gitlab-ci-multi-runner </code></pre>Double Check that Gitlab Runner is working</h2> $ gitlab-runner </code></pre> You should see output something like this: NAME: gitlab-runner - a GitLab Runner USAGE: gitlab-runner [global options] command [command options] [arguments...] VERSION: 1.11.0 (33af656) AUTHOR(S): GitLab Inc. <support@gitlab.com> COMMANDS: exec execute a build locally ... </code></pre> I've shortened the output considerably. Basically, you should see lots of help text explaining how this command works. If you don't see help text and see an error message instead, reach out to me on Twitter or in the comments, or consider reaching out to Gitlab for support. Run a Job with gitlab-runner</h2> Let's assume you have .gitlab-ci.yml file already committed that looks like this: image: node:latest my_special_tests: script: - npm install - npm test </code></pre> We can run the job my_special_tests</code> like so: $ cd path/to/project $ ls .gitlab-ci.yml .gitlab-ci.yml $ gitlab-runner exec docker my_special_tests </code></pre> The output of that command will look nearly identical to what you normally see on gitlab.com in your job page. Let's break down the above syntax a little bit: cd ...</code> - just make sure you're in the root directory of your project</li> ls .gitlab-ci.yml</code> - there should already be a .gitlab-ci.yml file in this directory</li> gitlab-runner exec docker tests</code> gitlab-runner</code> - this is main command that allows you to control the gitlab-runner. This is same code used in a remote environment to run your tests on gitlab.com and on your own instances of gitlab.</li> exec</code> - this subcommand allows us to work directly with our local git repo and execute builds without doing lots of fancy setup. It's quick and dirty -- exactly what we want for quick debugging</li> docker</code> - this means "use the docker executor to run this build". This is the preferred way to run builds. However if you need to run builds on MacOS or Windows, you'll probably want to replace with shell</code> instead. You just won't get many guarrantees about the environment your shell runs in.</li> my_special_tests</code> - this is the name of the test to run taken from the .gitlab-ci.yml file. If you had a job named 'foobar' you'd repalce my_special_tests</code> with foobar</code> instead.</li> </ul> </li> </ul> Gitlab Runner Exec Gotchas</h2> Even your local changes must be committed to git or they won't be available to gitlab-runner</li> The cache and artifacts in your .gitlab-ci.yml won't work with the gitlab-runner exec docker</code> command</li> </ul> Protips</h2> Run gitlab-runner locally any time you're initially setting up a new .gitlab-ci.yml file. You can catch typos in your shell scripting or find problems with your $PATH</code> in a few seconds or minutes instead of hours.</li> Leverage Docker - you'll be able to debug problems from a machine using the big 3 operating systems. You'll also be guarranteed an identical build in your remote CI.</li> Fallback to deploying from local machines if your cloud based CI goes down; just make sure to use a password manager like LastPass or 1Password to store deployment secrets, and document the procedure. This can be a really nice way to reduce a single point of failure.</li> </ul> Conclusion</h2> While CircleCI, TravisCI, and Jenkins have many fantastic features, its very difficult to get into a "flow" state when debugging them. This can be especially true if many other jobs are running at the same time. GitlabCI lets you do end-run around contention in CI and get results much faster. As a side benefit, when GitlabCI goes down, you can continue to run your deploy scripts locally by invoking Gitlab Runner directly. Setup Gitlab Runner right now and speed up your own debugging speed. Why I Use GitlabCI 2017-02-01T00:00:00+00:00 I've started moving all my projects away from Github in favor of Gitlab, and I haven't looked back. I should say upfront that I'm not affiliated with nor employed by Gitlab. I haven't been paid, prompted, or asked by Gitlab to write this post. And, as far as I know, Gitlab has no idea I've written about them. My favorite Gitlab Features</h2> Unlimited free closed projects, like Bitbucket</li> Painless Github importer</li> Remove source branch after merge to master automatically</li> Merge branch when tests pass automatically</li> Create a branch from a issue in website; auto-resolve issue when branch is merged</li> Integrated approach to software development makes it easier see what's happening in my project</li> Excellent issue tracker / basic project management</li> Integrated CI with excellent documentation</li> Proper docker support in CI (not a forked build of docker)</li> Builtin Docker image registry</li> Build environment can be any docker image</li> Parallel builds in CI</li> API for CI</li> CI Deployments track where your code is deployed to (I use this feature to render previews of my blog on S3 on a per branch basis)</li> </ul> Gitlab is more innovative</h2> Gitlab has pioneered a plethora of new features before Github, Bitbucket, Gogs or any of the other competitors in the space. Gitlab's commitment to open source product direction, incident response, and source code means that I have insight into the product roadmap as well as the ability to contribute my feedback and features. By comparison, Github has more-or-less copied a few of the best project management features but still lags behind Gitlab in terms of basic features like "Merge on Test Pass" and "Remove Source Branch on Merge" features. Github itself is 100% closed source and notorious for its lack of transparency in terms of product direction. Github eventually added the ability to rebase-merge, and create issue templates months and months after Gitlab. By choosing to use Gitlab, you're getting the best new features right away. GitlabCI is competitive</h2> Let's talk about GitlabCI. Here's a quick comparison (due to age poorly) between the major continuous integration and continuous deployment options in the space: CI/CD Service</th> GitlabCI</th> TravisCI</th> CircleCI</th> Codeship</th> DroneCI</th> Jenkins</th></tr></thead> VCS Support</td> git</td> git</td> git</td> git</td> git</td> any via plugin</td></tr> Github Support</td> no, but..</td> yes</td> yes</td> yes</td> Yes</td> plugin</td></tr> Gitlab Support</td> gitlab</td> no</td> no</td> yes</td> No</td> plugin</td></tr> Bitbucket Support</td> no, but..</td> no</td> yes</td> yes</td> Yes</td> plugin</td></tr> Open Source</td> Yes</td> Yes</td> No</td> No?</td> Yes</td> Yes</td></tr> Founded</td> 2011</td> 2011</td> 2011</td> 2011</td> ???</td> 2004 (as Hudson)</td></tr> Documentation Style</td> Reference</td> Tutorial</td> Tutorial</td> Tutorial</td> Reference</td> Tutorial</td></tr> Documentation Search</td> No</td> No</td> Yes</td> Yes</td> No</td> No</td></tr> Cloud Option</td> Yes</td> Yes</td> Yes</td> Yes</td> No</td> CloudBees</td></tr> On-premise Option</td> CE & EE</td> No</td> EE</td> No</td> Yes</td> Yes</td></tr> Local execution</td> Yes</td> Maybe...</td> No</td> Yes</td> Maybe...</td> Docker</td></tr> Debugging</td> Local</td> SSH</td> SSH</td> Local</td> Web</td> No</td></tr> Linux Test env</td> Yes</td> Yes</td> Yes</td> Yes</td> Yes</td> Yes</td></tr> OSX Test env</td> Not Cloud</td> Yes</td> Yes</td> Not Cloud</td> No</td> Yes</td></tr> Windows Test env</td> Not Cloud</td> No</td> No</td> Not Cloud</td> No</td> Yes</td></tr> Docker support</td> BEST EVER</td> Yes</td> Awful</td> Yes</td> Yes</td> Yes</td></tr> Enterprise Support</td> Yes</td> Yes</td> Yes</td> Yes</td> No</td> Probably</td></tr> Free Closed Source Projects</td> Yes</td> No</td> No</td> 1 project</td> On prem</td> On prem</td></tr> Build API</td> Yes</td> No</td> Yes</td> Yes</td> No</td> Yes</td></tr> Manual Test</td> Yes</td> No</td> No</td> No</td> No</td> No</td></tr> Source Controlled Config</td> Yes</td> Yes</td> Yes</td> Yes</td> No</td> No</td></tr> Secrets management</td> Web</td> YML & Web</td> Web</td> Web&docker</td> Web</td> Web</td></tr> Artifacts</td> Native</td> S3</td> Native</td> S3/SFTP</td> S3/Heroku</td> plugins</td></tr> Parallel Build "Steps"</td> Yes</td> Yes</td> Yes</td> Yes</td> No</td> No</td></tr> Parallel step rerun</td> No</td> Yes</td> No</td> No</td> No</td> No</td></tr> Parallel synchronization</td> Yes</td> No</td> Yes</td> Yes</td> No</td> No</td></tr> Build notifications</td> Yes</td> Yes</td> Yes</td> Yes</td> Yes</td> Yes</td></tr> Integrated Docker Registry</td> Yes</td> No</td> No</td> No</td> No</td> No</td></tr> Deployment targets</td> Yes</td> 3rd party</td> 3rd party</td> 3rd Party</td> 3rd party</td> plugins</td></tr> Badges</td> Yes</td> Yes</td> Yes</td> Yes</td> Yes</td> plugin</td></tr> Integrated Webhooks</td> Yes</td> No</td> No</td> Yes</td> No</td> No</td></tr> </tbody></table> One of the key differentiators above is the ability to run the CI locally. This is really really handy for two reasons: Increased fault tolerance: even if your remote CI infrastructure is down, you can still run builds and deployments.</li> Debugging: its potentially much faster to iterate on a broken test locally</li> </ul> Both GitlabCI and Codeship are designed to help with the second bullet point. GitlabCI also provides the increased fault tolerance that I often wanted when I relied on CircleCI. For me, GitlabCI is the most exciting CI option available, especially in the build-on-mac-and-deploy-to-linux scenario. Summary</h2> Gitlab unifies all (or most) of the different services you previously relied on when using Github, but is flexible enough to work with other systems (via triggers, API, and webhooks) or simply does a better job than Github + other services. If you're thinking about making the switch, I'd encourage you to try out the gitlab.com for your next side project or open source project. A Quick Intro to the NERDTree plugin for Vim 2016-08-04T00:00:00+00:00 NERDTree seems to be the de facto file explorer for Vim, but it's hard to find a good introduction to its quirks and some basic configurations without duckduckgo'ing tons of blog posts and reverse engineering individual .vimrc</code>s. This post tries to compile all the things I learned over several hours into an easily digestable summary. Show Hidden Files</h2> By default, NERDTree hides dot files. For many developers, this is a frustrating default setting that makes it hard to open and edit .gitignore, .travis.yml, and other really important dotfiles. I couldn't find any documentation about how to change the default in my .vimrc</code>, but here's how to do it: " .vimrc let NERDTreeShowHidden=1 </code></pre>Filter Out Custom Files and Directories</h2> The syntax is a little weird, so let's look at an example and then break it down: " .vimrc let NERDTreeIgnore=['.git$[[dir]]', '.swp'] </code></pre> Use the variable NERDTreeIgnore</code> to configure the filters</li> Specify an array of patterns to ignore, using [</code> and ]</code> to bookend the list of comma delimited patterns</li> Files should be specified as string literals (ex: '.swp'</code>)</li> Directories should be specified as string literals with the magic suffix $[[dir]]</code></li> </ul> I'm guessing that glob patterns are also supported, but I haven't experimented with that yet. Open a PR on this article if you have experience with globs! Update NERDTree's View of Files</h2> TL;DR Click on the NERDTree buffer</li> Press R</code> or r</code></li> </ul> Full story For performance reasons, NERDTree caches its view of the file system. This means that if you add, rename, move, or delete a file outside NERDTree, you won't be able to see that change reflected. Adding, Renaming, Moving, and Deleting Files and Directories</h2> Click on the NERDTree buffer</li> Press m</code></li> Pick the option you want</li> </ul> Conclusion</h2> This is all super basic NERDTree stuff, but I couldn't find a good summary elsewhere. What are you advanced tricks to get the most of NERDTree? Builtin Neovim Features Everyone Should Know 2016-07-19T00:00:00+00:00 For longtime (Neo)Vim enthusiasts, nothing here will blow your mind. However, if you've always felt like there are more builtin features of Vim you're missing out on, here's a handful of useful features I'd overlooked for years which don't require any plugins. Neovim's Builtin Terminal Emulator</h2> This is probably the most killer built-in upgrade that Neovim offers over standard Vim. Basically, Neovim exposes a new "terminal" mode that forwards all your keystrokes to an underlying terminal (which is handled by a cross-platform terminal emulator library). Compared to using tmux + Vim, Neovim's terminal editor lets you keep all your buffers (and terminal) in a single Vim instance. So your jump list, yanks, and global state are shared. For folks like me too lazy to learn tmux, I can cross that off my list. Starting Terminal Mode in the Current Buffer</h3> :terminal </code></pre> And then type anything that will put you into insert mode (a</code>, o</code>, i</code>, etc). Start Terminal Mode in a New Buffer</h3> If you'd rather split the current buffer and start a terminal in that, type this: :sp term://bash </code></pre> Then click or <C-W>h</code> into the new terminal buffer and use insert mode. Exiting Terminal Mode</h3> <C-\><C-n> </code></pre> Personally, I find that a bit too cumbersome, so I'm definitely going to remap that to something else. To learn more about terminal mode, see the official docs</a>. Deciphering Vim's Key Notation</h2> I found myself perpetually confused by the key notation I'd see everywhere online and in the builtin docs. Finally, I determined to search until I found an explanation. Here's a semi-official wikia entry</a> which does a great job of explaining. Here's the major takeaways: <C-...></code> means CTRL and ... at the same time. Ex: <C-k></code> = CTRL+k</li> <S-...></code> means Shift and ... at the same time.</li> <A-...></code> means Alt and ... at the same time.</li> <BS></code> is backspace (not profanity...silly I know).</li> <F1></code> is the F1 key. No shocker there, just mentioning it in case you can't easily access your function keys.</li> <ENTER></code> is the enter key.</li> </ul> Additionally, you can always escape a single key stroke in insert or normal mode (and possibly other modes too) by pressing CTRL-V first. For example, CTRL-V and ENTER becomes </code> which is the same as <ENTER></code> but a lot fewer keystrokes. Vim's Visual Blockwise Edit</h2> Seriously...how did I not know about this? How?! Basically, you can visually select a rectangular region of the current buffer and make the same change to all of them simultaneously. This is a really fast way to change a list of items that start with a single quote into a list of items that start with double quotes, or other tedious tasks which are difficult to change by using regex. Here's a quick example of visual blockwise edit in action: In that gif, I pressed CTRL-V</code> (sorry, using the mouse doesn't work by default) in insert mode to start visually selecting. Next I pressed l</code> three times. Then I pressed j</code> repeatedly to select down several rows. Instead of the selection wrapping around the end of the current line, a "block" of text is selected straight down. Any operator will be previewed on the first line but applied to all lines of the block selection. I pressed x</code> to delete the entire selection. Another idea that you might try is changing the whole selection with c</code> followed by ESC</code> after your changes. Search and Replace Within a Selection</h2> Use the cursor to select (or press v</code> and use motion commands) to select a block of text</li> Press :</code>. You should the statusline show :'<,'></code> which simply means the selection</li> Type :s/old/new/g</code> and press <ENTER></code></li> </ul> Vim's Ways of Moving</h2> Go to line (normal mode). Type the line number then capital G. Ex: 200G</code></li> Ignore case for only one search. Add \c</code> to the end (for "case"). Ex: /FOOBAR\c</code></li> Jump a paragraph forward }</code></li> Jump a section forward ]]</code></li> Jump back to a previous jump - CTRL-O</code></li> Jump forward to a previous jump - CTRL-I</code></li> </ul> A "jump" can be triggered by using /</code>, }</code>, ]]</code> and other forms of movement. See :help jumps</code>. Conclusion</h2> Writing this article made me realize how much of Vim I'm not taking advantage of, and that it always pays to invest a little more time and effort into your tools. What's your favorite builtin features of (neo)vim? I'd love to hear in the comments. Better yet, open a PR against this article with your corrections and additions. Golang Testing stdlib Errors 2016-04-05T00:00:00+00:00 It can be tough to get anywhere close to 100% code coverage when funcs in the "stdlib" can't be forced to error. This post takes you through several of the common funcs built in to golang and shows you how to trigger an error to help you bump up your code coverage. This list is short and skews heavily toward http, where I spend most of my time. encoding/json.Marshal</h2> Marshal any func</code>, chan</code>, or complex</code>: serialized, err := json.Marshal(func(){}) </code></pre> See in the playground</a> net/url.Parse()</h2> Pass invalid hex strings: u, err := url.Parse("%zzzzz") </code></pre> See in the playground</a> See unit tests for unescape</a> net/http.NewRequest()</h2> The url in NewRequest()</code> also goes through url.Parse()</code>, so we can reuse our url breaking tricks. r, err := http.NewRequest("GET","/%zz", nil) </code></pre> See in the playground</a> net/http.ListenAndServe()</h2> We can pass an invalid port number to trip up ListenAndServe()</code>. Full working example this time since this won't run in the playground. package main import ( "net/http" "log" ) func main() { h := func(w http.ResponseWriter, r http.Request) { w.Write([]byte(`Hello World`)) } http.HandleFunc("/", h) log.Fatal(http.ListenAndServe(":9999999", nil)) } </code></pre> If choosing the port is not an option, we can cheat on our unit test, and block that port on the host. However, the stdlib doesn't support this in any form, so you'd have to pick a stoppable http server like tylerb/graceful</a>, like so: package main import ( "fmt" "net/http" "gopkg.in/tylerb/graceful.v1" ) func main() { // First block the port we care about h1 := func(w http.ResponseWriter, r http.Request) { w.Write([]bytes(`Blocked`)) } srv := &graceful.Server{ Timeout: 5 time.Second, Server: &http.Server{ Addr: ":1234", Handler: http.HandlerFunc(h1) }, } srv.ListenAndServe() // Now try to use the blocked port h2 := func(w http.ResponseWriter, r http.Request) { w.Write([]bytes(`Blocked`)) } fmt.Println(http.ListenAndServe(":1234", http.HandlerFunc(h2)) // Stop the blocking server srv.Stop() } </code></pre>net/http.Client</h2> Often, you want a way to force a connection error to simulate problems with some API you're consuming as part of your service. Golang's stdlib provides the RoundTripper</code> interface to allow us to customize interaction with the network. package main import ( "errors" "fmt" "net/http" ) // Force errors or responses of our choosing to be returned type mock struct {} func (s mock) RoundTrip(r http.Request) (http.Response, error) { return nil, errors.New("Unable to connect") } func main() { client := &http.Client{ &mockRoundTripper{}, nil, nil, 0 } resp, err := client.Get("https://www.google.com") fmt.Println("resp=", resp) fmt.Println("err=", err) } </code></pre>What Else?</h2> If you run into a golang func that you can't break, post it in the comments here or tweet it to me. Essentials for Cloud Deployment 2016-01-05T00:00:00+00:00 The most critical thing a deployment pipeline can do is instill confidence in your team. Build your pipeline on a bedrock of credible integration tests. Produce those results in an expedient manner, and deliver your software to customers immediately. When your developers believe in their deployment pipeline system, they can achieve great things. Credible Integration Tests</h2> Unit tests are not worth very much. Only tests which walk through the exact scenarios your customers will actually use on their actual data in your network can tell you if you've done your work correctly. In a prior life, my team developed an axiom: Master is always deployable </blockquote> This rule led us inexorably to the following rule: Only tested code can be merged to master </blockquote> But to satisfy this condition, arbitrary branches must be deployable to a "real" environment. Most often, production incidents occurred because our "real" environments were not real enough. For instance, I once personally wrote a feature to validate account data before saving it to the database. However, old customer accounts were "invalid" according to my feature, and we received many complaints from customers unable to login days later. How could this happen? I had written unit tests, integration tests, test fixtures, code review, and manually QA'd the work in a remote environment. The problem was that I had not tested using the production database. I could tell many similar stories of trying to migrate logs files to new block devices, "optimizing" database queries, or adding load balancers and breaking deployments. The point is your tests can't save you from catastrophic disasters if you can't 100% trust them. 100% Trusting Your Tests</h3> Here's a(n incomplete) list of things that are prerequisites for trusting your integration tests: Remote servers</li> Production traffic</li> Real data</li> Real network topology</li> Happy and sad path testing</li> "Production" mode</li> </ul> Remote Servers + Real Network Topology</h4> Docker is the best way to guarrantee the local environment for your application is identical between development and production (and you should use it). However, docker can't guarrantee your DNS is configured correctly or that the VPN connection works properly. It can't guarrantee your changes won't bork the service orchestration nor service discovery, nor any of a thousand other things that just couldn't matter but somehow do. At least in AWS, different instance types can have radically different features. For example, an m3.medium will only run in a virtual private cloud which requires some serious networking knowledge, whereas a m1.medium works just fine in the stupidly-easy public cloud. A spot instance can disappear at any time or fail to launch if the spot price suddenly spikes, whereas an on-demand instance is guarranteed to boot up and stick around as long as you want it. These are just a few obvious differences. Make sure you use the same instance types to run tests against your feature branches and bug fixes that your production stack uses. Production Traffic + Real Data</h4> Honestly, real data is most important with NoSQL databases where documents can be anything. If you're using Postgres with a Rails App, this may be less important. Duplicating production traffic is hard, but the kind of confidence it can give is unparallelled. You can instrument optimizations (especially around your database or cache) and know it will make your existing infrastructure more scalable without ending your startup. There's just no other way to get those guarrantees. Happy + Sad Path Testing</h4> Testing failure scenarios is hard and counterintuitive. Which is why it causes so many production incidents. If there is a value to writing unit tests, I would say that seeing code coverage visually present all the code paths we missed is the highest one. Some testing frameworks (like rspec on its own) can make it feel hard to test lots of failure scenarios. However, I've personally found that cucumber really makes it easy to create tons of integration tests. The examples</code> and tables make it easy to add subtly different versions of the same test. The step definitions decouple specific tests from reusable code blocks in a super elegant fashion. Those and a million other things, make cucumber my new goto tool for less-painful integration testing. "Production" Mode</h4> This is just wrong: if (environment == "production") { // Something untestable that will haunt your dreams } else { // A sirensong luring you into false security } </code></pre> Look, nothing interesting or important should ever happen in production that you haven't properly tested before. If you hardcode specific environment detection, you by definition can't test something interesting. This leads to surprising and harmful situations. Where should this kind of setting go? Anywhere else: Environment Variables</li> Config file</li> Database</li> Cloud-config user-data</li> </ul> I've seen several great examples of a config</code> abstraction which can be queried for environment specific variables. So, something like this instead: var databasePassword = config.get('databasePassword'); </code></pre> This code snippet is better than the conditional above because it guarrantees similarity between your local machine, your remote test environment, and your production server. Only Fast Tests</h2> If your integration tests take more than time than it takes to walk to the water cooler and back, your developers are going stop working and drift to hacker news for the next hour. Wait? What was I working on again... Every integration test scenario should be able to run in any order so that you can parallelize the shit out of your tests. Write a lot of tests, but run them all at once across multiple machines/containers/vms/whatever. Don't settle for anything less than awesomely fast. The secret to keeping your tests fast is to chip away at each subsequent bottleneck every sprint. Deliver Immediately</h2> Why does it take so long to deploy changes as your cluster gets bigger and bigger? Because you're rolling instances onesy-twosy. Rotating instances individually is slow and error prone. What happens if you get halfway through a deployment and one instance never comes back online? [Gulp.] There is a right answer here. Bring up all new nodes at once while the old nodes are still running. The new instances will come online at basically the same time. When they do, split production between old and new nodes until your smoke tests pass. Then, drop the all the old instances at once. If some of your new nodes don't come online or fail smoke tests, just abort the automated deployment and then manually back off production traffic. Your old nodes will still be humming along in the background. This will take you from an O(n) deployment speed to a O(1) deployment speed. Deployment Pipelines as Emergency Preparedness</h3> Mistakes happen. Apps break. We can't ever totally eradicate it, but having a fast deployment pipeline means that we can respond more quickly and minimize disruptions to our customers. Moreever, we can delight by spending more time developing fixes and features and less time waiting around. Inspiring Deployment</h2> Any time something "just works," it's truly a thing of beauty. If you're a web developer, you need a fast reliable way to get your work into your customers hands. Tell me your secrets for getting more out of your deployment pipeline, or tell me why I'm wrong above. Testing AWS Go SDK Integrations 2015-10-25T00:00:00+00:00 When programming in Golang, the Mockery project painlessly generates Testify mocks from any interface. But very rarely (and specifically in the AWS Go SDK), you run into situations that Testify doesn't handle well. If you think that Testify has failed you, then prepare to enter... a minefield of horrors! </a> The Post-Testify Minefield of Horrors</h2> This "Post-Testify Minefield of Horrors" I speak of is a simple trilemma. You must decide between one of the following tradeoffs: Don't unit test this code -- what could go wrong?</li> Abandon Testify for this package -- hand write ALL the mocks!</li> Keep using Testify -- rewrite the mocks EVERY time your regenerate your mocks.</li> </ol> None of these are great options. What if there was a way to use Testify mocks 99.99999% of the time, but handcraft only the special cases it doesn't cover? That's what I aim to offer you by the end of this post -- a way through this minefield. You Lie! Testify Is All I Need</h3> For AWS APIs which aren't paginated, Testify works swimmingly, and Mockery does a fantastic job of writing all the tedious mocks for us. However, as a prime example where Testify seems to break down, consider the DynamoDB package with the AWS Go SDK. Let's say you want to query one of your DynamoDB tables, but the total amount of data you want to retrieve could become greater than 1 MB. According to the documentation: If you query or scan for specific attributes that match values that amount to more than 1 MB of data, you'll need to perform another Query or Scan request for the next 1 MB of data. </blockquote> To ensure that your application retrieves all the query results, you need to call DynamoDB.Query()</code> repeatedly with different inputs and outputs. That's all very simple, but to unit test this code you'll need a mock implementation of DynamoDB.Query()</code> which can return different values on each call. Although Mockery shows some examples of this in their README, returning dynamic values seems not to work at time of writing (late October 2015). So, this ideal solution I've promised you should both: Use Testify mocks generated painlessly by Mockery for most of the AWS Go SDK</li> Allow us to handcraft select pieces of highly dynamic AWS Go SDK code</li> </ul> So how can we achieve both? The Solution: Separate Interfaces</h2> </a> We'll write one interface that acts as a contract between our code and AWS Go SDK in general, and we'll let Mockery generate mocks for this interface. Then, we write a very small dedicated interface for the handcrafted mocks. Any of the non-paginated AWS APIs we use in our code will go through the general interface, but our application will Query()</code> only through the handcrafted interface. Since Mockery puts its generated code in a subdirectory ./mocks</code>, we can ensure our handcrafted mocks will survive by keeping them out of ./mocks</code>. The Big, General Interface Interface</h3> Since DynamoDBer</code> stutters, let's just call it DBer</code>. The general interface DBer</code> should look something like this: import "github.com/aws/aws-go-sdk/services/dynamodb" type DBer interface { GetItem (/argument/) (/return values/) DeleteItem (/argument/) (/return values/) UpdateItem (/argument/) (/return values/) // ... } var _ DBer = (dynamodb.DynamoDB)(nil) </code></pre> Then to call any method in AWS Go SDK, we always make sure that method has been added to DBer</code> and use the DBer</code> interface in our code, something like this: func Get(id int) (Item, error) { var db DBer = dynamodb.NewDynamoDB() item, error := db.GetItem(id) return item, error } </code></pre> Don't try to compile that, but you get the gist. Now, to get all our mock objects for free, we'll install Mockery. Then, we'll cd into the code directory and run Mockery: $ go get github.com/vektra/mockery $ mockery . -name=DBer </code></pre> At the end of the process, I'll have a shiny new directoy ./mocks</code> containing all the Testify mocks for me. Sweet! Okay, that's the big DBer</code> interface. Let's move on to the other interface. </a> The Small, Handcrafted Queryer Interface</h3> import ( "github.com/aws/aws-go-sdk/services/dynamodb" ) type Queryer interface { Query(QueryInput) (QueryOutput, error) } var _ Queryer = (dynamodb.DynamoDB)(nil) </code></pre> A Queryer</code> only needs to implement the Query()</code> method already in the DynamoDB package. For good measure, we create a nil and ignored DynamoDB struct object as a Queryer</code> so that the compiler will verify that we have the method signature of Queryer.Query()</code> matching DynamoDB.Query()</code>. We want to make sure the compiler catches any obvious mistakes (like failing to satisfy an interface) before we ship this code into production. Otherwise, we might theoretically some day update the AWS Go SDK package, have the code compile, and ship this out into production only to discover failures too late. Wrapping Up Queryer in a Useful Func</h3> Let's assume your application should always retrieve all the results, we'll want to create a wrapper method to DRY up and encapsulate all the logic necessary to retrieve all pages of results. func paginatedQueryImpl(q Queryer, i QueryInput) ([]items, error) { // ... } var paginatedQuery = paginatedQueryImpl </code></pre> Let's unpack this a bit. Why both a paginatedQueryImpl()</code> and paginatedQuery()</code>? The idea here is that we want to easily mock out the method that our application will actually use, so I've created a mutable variable paginatedQueryImpl()</code>. This allows me to change the implementation on the fly in unit tests by just assigning a new lambda function. However, for my unit tests on paginatedQuery()</code> itself, I need to ensure that I'm testing the actual implementation. For that kind of assurance, I need to declare a function with the real implementation. Then, in my unit tests I can simply reassign the real implementation to my paginatedQuery</code> variable as a setup step. Then, I need a way to test the interaction between paginatedQueryImpl</code> and DynamoDB, and that's precisely why it takes a Queryer</code> argument. This allows us to pass in either real DynamoDB</code> objects or our handcrafted mocks. The handcrafted mock will simulate all the various test cases that we care about without needing to send any requests over the wire or even have an AWS account. The implementation of paginatedQuery()</code> is left as an exercise to the reader, since testing is the primary focus of this post. Handcrafting the Mock Queryer</h3> import ( "github.com/aws/aws-go-sdk/services/dynamodb" ) type MockQueryer struct { QueryFunc func(QueryInput) (QueryOutput, error) } func (m MockQueryer) Query(i QueryInput) (QueryOutput, error) { return m.QueryFunc(i) } </code></pre> There's a few salient points to discuss here: We need the QueryFunc</code> field to be a lambda function, because we want to replace it with different implementations in each test case. We don't want to define a different struct type for every test -- that's just too much boilerplate code.</li> Because the QueryFunc()</code> and Query()</code> both have the same signature, we can simply passthrough the arguments and return values from the Query()</code> to QueryFunc()</code> and get compile-time checks that we defined our lambdas correctly in the tests.</li> </ul> Let's Write Some Example Unit Tests</h3> The F# community's writings about property-based testing</a> have had a profound impact on me. Whenever appropriate, I like to include some randomness in my tests to ensure that a whole range of similar but unpredictable values are tested to prevent an incomplete implementation from passing. import ( "math/rand" "github.com/stretchr/testify/suite" ) // Define a Testify suite `MySuite` here and run it. func (s MySuite) TestPaginateQuery_NQueryResultPages(t Testing.T) { minPages := 3 maxPages := 50 itemsPerPage := 10 totalPages := rand.Int31n(maxPages - minPages) + minPages totalItems := totalPages itemsPerPage c := 0 keys := rand.Perm(int(totalPages)) mq := &MockQueryer { QueryFunc: func(i QueryInput) (QueryOutput, error) { // Run assertions on the arguments to QueryFunc() actual := i.ExclusiveStartKey expected := keys[c-1] if c > 0 { s.Equal(i.ExclusiveStartKey, keys[c-1]) } // Generate the QueryOutput with correct LastKeyEvaluated output := makeQueryOutput(c, itemsPerPage, totalPages) c++ return &output, nil } } items, err := paginatedQuery(&mq, &dynamodb.QueryInput{}) s.Len(items, totalItems) s.Nil(err) } </code></pre> In our test method TestPaginateQuery_NQueryResultPages</code>, the first five lines randomly pick how many pages of results and total items should exist when this test runs. The next several lines of code create call counter c</code> and create a new MockQueryer</code> object mq</code>. This object is passed into our the method we want to test PaginateQuery()</code>. Now, we have complete control inside the QueryFunc method to calculate whatever return values we want AND to run assertions on the arguments to this method. My test leaves a lot to be desired (and probably doesn't even compile), but hopefully it can serve as an inspiration. Testing Code that Uses PaginateQuery()</h2> I believe the clearest and easiest way to test code that invokes paginateQuery()</code> is using the mutable/immutable functions I've defined earlier. This allows us to stub replace the func during our tests and gives us unfettered access to control in the innerworkings of our code. Let's see a short example: // retrieve_everything.go func RetrieveEverything() ([]item, error) { dynamo := dynamodb.NewDynamoDB() return paginatedQuery(&dynamo, &QueryInput{}) } // retrieve_everything_test.go func (s MySuite) TestRetrieveEverytingPropagatesErrors() { paginatedQuery = func(_ Queryer, _ *QueryInput) ([]item, error) { return nil, errors.New("Dynamo Broke") } items, err := RetrieveEverything() s.EqualError(err, "Dynamo Broke") s.Nil(items) } </code></pre> In a similar fashion, we can replace paginatedQuery()</code> with mocks that will return nil errors, improperly formatted data, a slice of results, an empty slice, or any other scenario that could conceivable happen. Damn Your Smoke and Mirrors Lambdas!</h3> The downside of this approach is that any part of your codebase could overwrite the func pointed to by paginatedQuery()</code> which seems a bit unsafe. If any unit tests replace it with a mock, you'll need to reset it to paginatedQueryImpl()</code> as discussed earlier. However if global mutable lambdas makes you uncomfortable, you can use Mockery to generate a mock for DynamoDB.Query()</code> and indirectly control the output of paginatedQuery()</code>. If you go that route, just don't use a package variable. Instead, make paginatedQuery</code> a proper immutable func in its own right. As far as I can tell, which strategy you use is entirely a matter of preference. Taking it Further</h2> The Mockery Project</a> is well worth a look if you haven't seen it before.</li> Nearly everything I've said here applies equally to any other paginated function in the AWS Go SDK (and that's a lot of functions). If you use this sort of technique a lot, it might make sense to write a code generator that can DRY up repeated logic and tests around this functions.</li> If you're curious about DynamoDB, see the AWS documentation for Query</a></li> Consider using AWS's DynamoDB QueryPages() method</a> instead of Query()</code> in your own code since it handles lots of pagination messiness for you.</li> I omitted the definition of the MySuite</code> struct in one of the examples above for brevity. See the Testify suites godoc</a> if you need a refresher.</li> If you need to rerun just one test several times while you're debugging or writing tests, use go test ./path/to/package -run TestFunc</code> where "TestFunc" is the name of the test that go knows how to run (ex: the func that starts your Testify suite).</li> For integration tests, I recommend trying AWS's local-dynamo jar</a>. It seems to be maintained and reasonably performant as a drop-in replacement for dynamodb that runs on your own machine or CI build environment.</li> </ul> Conclusion</h2> Golang is a fantastic language for writing unit tests, but it sometimes requires some creativity to test all the code paths you care about. Consider using smaller interfaces to mock out isolated complex dependencies that Testify can't handle well. Special Thanks to Julian Cooper for all the ideas about lambdas and small interfaces presented here, to Phil Cluff for allowing me to write about my work, and my wife Emilie Fisher-Fleig for the insightful comments on my drafts. Effective Commit Messages 2015-05-25T00:00:00+00:00 There are a variety of ways to include documentation in your project. I want to focus on commit messages as a vital clue in debugging and refactoring your code. Although I use git exclusively at this point, similar techniques will work for mercurial (and svn to a lesser extent). Strengths of Commit Messages</h2> The great thing about using git commit messages is that they persist exactly the same length of time as the code they describe. This is a huge advantage compared to comments in the source code. When that code changes or disappears, so do the commit messages. Using commit messages as documentation ensures always up to date documentation. A commit has a timestamp and an author associated with it. Additionally, a commit has a position in git history. For teams adopting Github (or similar services), you can often look forward from a particular commit in git history to determine which pull request and agile story this line of code came from. You can examine adjacent commits for extra clues about related changes. If you get really confused by a message, you know exactly who wrote that message and you can talk to that person if they still work at your company. In git and mercurial, a commit preserves a context, because it is simply a patch of the previous commit. You can see all the other changes that happened at the same time, which can help ask that elusive why question: "Why was this code written?" Often, simply reading through the other code written at the same time helps me wrap understand a particular line of code. You don't have to adopt any new software, database, or programming philosophy. If you're using version control software already, this information is already there. Whether or not you leverage all of this metadata is up to you. Comparison to Source Comments</h2> The problem with comments is that they can remain in the codebase a long time after the code they originally described is removed or altered. This means that the context can be ambiguous. A related problem is that changes to the code are not always accompanied with changes to the comments, leading to inaccurate or out of date information. The main strength of comments versus commit messages is that comments are visible to readers of the source code. In my opinion, comments are best for situations in which there is some hack that is really important to understand and preserve. Any time I include a hack in source code because it seemed necessary, I leave a comment in the source code. Serious gotchas also deserve to be included in the source code directly as well. However, like any code, comments themselves add a maintenance burden. I tend to use commit messages anywhere I can. Retrieving a Commit Message</h2> Commit messages are harder to find than comments. Let's look at how to find the commit that a line of code belongs to. In Atom</h3> This screenshot shows using the git-blame package</a> for Atom. In Eclipse</h3> This screenshot comes from EclipseGit package</a>. In Sublime</h3> This image comes from an ancient forum post</a>, but it gives some idea of what to expect. In the Shell</h3> Use git blame</code></a>, like this: $ git blame -L22,25 src/model/resource.js </code></pre> -L stands for lines, in this case lines 22 to 25</li> src/model/resource.js is the specific file you want to look into</li> </ul> This code displays the shortened commit hash next to the contents of those lines from the file. Mercurial has a nearly identical command hg blame</code>. To actually see the corresponding log message, use: $ git show abcde12345 </code></pre> where abcde12345</code> is the actual commit hash from git blame. If you know of a way to get the commit message from a specific file and line number in a single command, please tell me in the comments! Writing the Commit Message Title</h2> If you use Github, you'll also see that Github will use the first 50 characters of the first line of your commit message as a heading for that commit (the second line needs to be blank): ...or as the title for a single commit PR: If you use vim to edit your commit message, the first 50 characters are one color, and anything beyond 50 will turn another color. Using that visual clue helps you communicate the purpose of a commit to the rest of the team at a glance. However, if you ignore this 50 character cutoff, commmit messages will be cut up and difficult to read, like this: Categorizing the Commit</h3> Another helpful convention I sometimes follow is starting the commit title with the kind of change I'm making: (fix) - no new feature, simply a fix for a bug in existing code</li> (feat) - new functionality introduced, possibly tests and documentation as well</li> (refactor) - reimplements internals</li> (doc) - updates to a README or comments</li> (test) - adding or fixing a test</li> (nit) - changes to whitespace, code style, or something else super trivial</li> </ul> Helpful Commit Message Bodies</h2> I like to use everything after the second line of a commit message to describe: what problem each commit is addressing</li> what subtle gotchas are lurking beneath the surface</li> why I choose this implementation over another way of writing the code</li> the issue number for later cross-referencing</li> </ul> I often use git bisect</code></a> to debug tricky but reproducible errors, and I like to think of commit messages as clues to my future self during a bug-hunt six months from now. But git bisect</code> is a subject for another post. Example Commit Message Body</h3> change all app.user.getUploadedFilesData() to get files metadata to asynchronous operations </blockquote> Not great -- why did we change to asynchronous operations? Also, this message is too long for github. It will be seen in PR titles or commit summaries as change all app.user.getUploadedFilesData() to g...</code>. That tells me almost nothing about what change happened or why! Let's see if we can improve things: retrieve user uploads from db for persistence </blockquote> This explains, at a glance, why this commit was made. Uploaded files are stored in the database now, why? For better persistence. Okay, so that's really important, and it provides me a lot more clues about the bug I'm trying to fix. But this doesn't mention anything about asynchronous operations... how can we fit that onto the first line? We can't. But we can provide more in-depth discussion of the details of the changes in lines 3 and later in the commit message. Let's take a look: retrieve user uploads from db for persistence Because this operation requires db access, all calls to app.user.getUploadedFilesData() are now asynchronous calls. We had to start using the database because all metadata (like filesize) was lost each time we restart the app. </blockquote> Better, but why didn't the author just grab this info from the file system? Let's try one more improvement: (feat) retrieve user uploads from db for persistence Because this operation requires db access, all calls to app.user.getUploadedFilesData() are now asynchronous calls. We had to start using the database because all metadata (like filesize) was lost each time we restart the app. Since we use clusters of machines, not all machines in the cluster have all files all the time. We use the database so that whichever machine in the cluster handles the request can access the file data. </blockquote> Perfect! We have a one line summary in github, we have a detailed explanation of why app.user.getUploadedFilesData() is now async, and we know why some easier option wasn't used. There's probably still plenty of room for improvement, but this is a great foundation. Example Trivial Commit Message</h3> Not all commits need extended explanations. Let's look an example for a change that's relatively trivial: fix colors </blockquote> That gives us some idea of what's happening, but it doesn't answer the why question. Let's see if we can improve this message a bit: matched header color to brand guide </blockquote> Much better! Now we can tell at a glance why this change was made and which kinds of colors were changed. But we still don't know where to find more info in our issue tracker. Let's make another change: (nit) matched header color to brand guide [203] </blockquote> This is a great message. We can go to issue 203 if we ever need to learn more about who wanted this work done, or why this shade of green was chosen instead of the actual shade specified in the brand guide. Summary</h2> Commit messages can be a superior form of documentation for a team that adopts them. There's nothing new or extra you need to do to take advantage of source control history today, but there's a lot of advantages to carefully describing the work you do in your commit messages. Special Thanks to Rex Vokey</a> for suggesting more images, Jeff Ballard</a> for broadening my horizons, Blain Sadler</a> for support, Matt McClure</a> for the inspiration, and my stunning wife for providing extensive feedback on this article. Setting up Istanbul with Jasmine on NodeJS 2015-04-17T00:00:00+00:00 Istanbul is perhaps the best test coverage tool for NodeJS. Istanbul's metrics include lines of code, functions, and branches; and it generates reports that color each line to indicate full, partial, or no coverage. I also love that the reports break down test coverage per directory and file, which I use to identify blind spots. While Istanbul can work with any testing framework, I focus on NodeJS and Jasmine, but much of the background information is relevant for other configurations. How Istanbul Works</h2> Istanbul is not a static code analysis tool, but rather it observes running code. It works in two distinct steps which are important to understand: Instrument the code</li> Run the tests against the new code</li> </ul> Instrument the Code</h3> By "instrument", we mean create a functioning mock of the source code, while still invoking the real implementation. It's much like using Jasmine's spyOn().andCallThrough()</code>. Effectively Istanbul is able to check whether or a given unit of code has executed, much like using Jasmine's expect(spy).toHaveBeenCalled()</code>. The main difference is that the instrumented code will become files rather than in memory objects. The configuration for Istanbul must point out which files to instrument. Perhaps the biggest gotcha here is instrumenting the tests. You don't want to do that for reasons that will become clear later on. The main thing to understand is that Istanbul needs to spy on your application, not your tests. Run the Tests</h3> Now that you have instrumented code, have Jasmine run it's tests against the instrumented code. This means that Istanbul can determine exactly which units of code executed during the test run. Our configuration will also need to specify how to run the tests. Once the test run is finished, Istanbul will output all the reports and display some summary metrics in the console. Configuration directs Istanbul to use one of several formats for the reports. Show Me The Code!</h2> Enough chitter-chatter. Here's how I setup things in my project: $ cd /path/to/project $ npm install --save istanbul jasmine-node $ mkdir -p test </code></pre> Istanbul and Jasmine-node are built as command line tools, thus normally they would be globally installed. Notice I'm not doing that. In my experience, global packages are fundamentally wrong headed and are the biggest disparity between dev and production environments. Fortunately, npm makes using non-global command tools easy. Add something like the following to your package.json file: { "scripts": { "test": "istanbul cover --include-all-sources jasmine-node test" } } </code></pre> Put all your tests for jasmine-node into the test directory. To run your test suite AND get test coverage do this: $ npm test </code></pre> Running npm test</code> is really a shortcut for npm run test</code>, which is the test</code> command we created in package.json. Although the istanbul</code> executable isn't normally available in the shell (because we didn't install it globally), npm makes it available inside the scripts listed in package.json. None of this should be suprising -- it's not specific to istanbul. Let's dig into the istanbul specific bits a little more: the --include-all-sources</code> flag causes istanbul to recurse through directories for more source files to instrument</li> everything after istanbul cover [flags]</code> is the test command (in this case jasmine-node test</code>)</li> by default, istanbul ignores the test</code> and tests</code> directory. If we had used spec</code> instead, we'd have to configure istanbul to ignore that directory</li> by default, istanbul creates reports in a directory coverage</code> under the current working directory</li> </ul> There's a lot more configuration options which I'm not going into here. If you want to explore them, try this: cd /path/to/project ./node_modules/.bin/istanbul help config </code></pre>Design Considerations</h2> There's lots of choices of you can make while setting up Istanbul, and a few problems to avoid. Git Ignore Coverage Reports</h3> The coverage reports are generated from your source, but you don't want to ship that into production. Add that directory to your gitingore. Gotcha: Instrumenting the Tests</h3> Let's revisit the gotcha mentioned above -- accidentally instrumenting your tests. If only your tests are instrumented, Istanbul will report 100% coverage. You'll slap yourself on the back -- 100%! We did it! Then you'll realize something's off. If you instrument your source code along with the tests, all metrics will skew upwards proportional to size of your tests. Not to be a broken record, here, but Istanbul is mocking out your entire application to observe what your tests are covering. If you accidentally have Istanbul mock out your tests, Istanbul will notice that all of your tests have run, which is not very helpful. To help keep tests and source code distinct for Istanbul, I recommend using a separate top level directory for tests and source code in your project. Running the Tests Once or Twice</h3> In my limited experience, running Jasmine against the instrumented code produces the same results as running Jasmine against the original source code. If Jasmine fails the test run, its process returns a non-zero exit code. Istanbul's process then passes through a non-zero exit code as well if Jasmine fails. Thus, however you run your tests (npm, grunt, gulp, Circle CI, Travis, etc), you can run the Jasmine tests once and get the results faster. If you don't trust the results of the instrumented code, the other choice would be to run the Jasmine tests twice -- once against the source code and once against the instrumented code. One or Many Coverages</h3> Not all tests are created equal. You might want to have integration tests as well as unit tests. Think hard about to categorize each kind of test you have (or want to have). You might consider having Istanbul check the unit test coverage separately from the integration test. Here's one way you might do that: { "scripts": { "test": "npm run unit && npm run integration", "unit": "istanbul cover --include-all-sources jasmine-node test/unit", "integration": "istanbul cover --include-all-sources jasmine-node test/integration" } } </code></pre> While it could be helpful to setup separate coverage checks, you won't have insight into the overall test coverage on the codebase as a whole. If you do decide to break test coverage into separate catgories, I also recommend having an overall coverage report generated. Handlebars Considered Harmful 2015-03-02T00:00:00+00:00 TL;DR: Handlebars is a DSL builder in disguise tempting you to create an undebuggable mess. If you minimize helpers and keep all procedural code outside handlebars, you can safely keep it in your toolbelt. DO try this at home, kids</h2> When I wrote this blog post back in 2014, I built a stupidly simple and deliberately buggy handlebars node app for this blog post. In the intervening years, I deleted this messy code on accident. All the error messages and code samples come directly from this repo. It all started so innocently...</h2> You're building a nodejs web application to track fitness, so of course you're going to use handlebars. It's what we nodejs developers do. Like all good startups, you move fast and break things. "Red, green, refactor" is your freaking middle name. The code you write may not be poetry, but you make shit happen. ...Until everything went terribly wrong</h2> One day, you receive an urgent bug report from the CEO that there's an icky error message every time he logs in. After verifying this issue doesn't happen for your user</a>, you can reproduce this problem by running the source code on your machine and trying to login as the CEO</a>. Here's the error: TypeError: Cannot read property 'activityStyle' of undefined at Handlebars.registerHelper.(anonymous function) (/var/app/helpers.js:45:47) at Object.eval (eval at <anonymous> (/var/app/node_modules/handlebars/dist/cjs/handlebars/compiler/javascript-compiler.js:209:23), <anonymous>:5:125) at Object.ret (/var/app/node_modules/handlebars/dist/cjs/handlebars/runtime.js:144:30) at Object.ret [as summaryActivity] (/var/app/node_modules/handlebars/dist/cjs/handlebars/compiler/compiler.js:462:21) at null.<anonymous> (/var/app/helpers.js:17:52) at Object.eval (eval at <anonymous> (/var/app/node_modules/handlebars/dist/cjs/handlebars/compiler/javascript-compiler.js:209:23), <anonymous>:9:76) at Object.prog [as fn] (/var/app/node_modules/handlebars/dist/cjs/handlebars/runtime.js:178:15) at Handlebars.registerHelper.url (/var/app/helpers.js:22:17) at Object.eval (eval at <anonymous> (/var/app/node_modules/handlebars/dist/cjs/handlebars/compiler/javascript-compiler.js:209:23), <anonymous>:4:83) at Object.prog [as fn] (/var/app/node_modules/handlebars/dist/cjs/handlebars/runtime.js:178:15) </code></pre>A Useless Stack Trace</h3> The only connection between the stack trace above and our code is that first line. Every other line in the stack trace is from Handlebars' compiler or runtime eval</code>ing anonymous functions. We have no idea what the context was that triggered the error in helpers.js. Hopefully it's just an obvious syntax error in helper.js line 45. It's interesting files names for a template package though, right? Weirdly, "compiler" and "runtime" remind me of something awfully familiar from my computer science classes... Enough daydreaming, let's see what's happening in helpers.js: Handlebars.registerHelper('setActivityStyleFromUser', function(user, options) { if (this.activityStyle == null && user.prefs.activityStyle) { this.activityStyle = user.prefs.activityStyle; } return options.fn(this); }); </code></pre> Damn! There's 2 usages of activityStyle</code> on line 45 that look a bit suspect. Never fear -- we have the {% raw %}{{log}}{% endraw %}</code> helper to show use what's going on. Phew! In the console, we see that the user.prefs.activityStyle</code> is what's giving us trouble. A plethora of confusing grep results</h3> But, how did we get here? Why is only the CEO crashing on login? If we rewrite this helper, what might we break elsewhere? Time to search through all the code with grep to find where setActivityStyleFromUser</code> is being used. Hopefully there's only one code path to this particular helper; otherwise, it's going to be a long night. bryce@server:/var/app $ grep -ir setActivityStyleFromUser . ./helpers.js:Handlebars.registerHelper('setActivityStyleFromUser', function(user, options) { ./components/burnIt.hbs: {{#setActivityStyleFromUser user}} ./components/burnIt.hbs: {{/setActivityStyleFromUser}} ./components/summaryActivity.hbs: {{#setActivityStyleFromUser user}} ./components/summaryActivity.hbs: {{/setActivityStyleFromUser}} {% endraw %}``` We've read through that first result, where the handlebar helper is being registered, so we can ignore that one. Both burnIt.hbs and summmaryActivity.hbs are possible places this is being used. Double damn! Two code paths possibly leading to the original error message, but we don't know which one is giving us trouble for sure. Let's look at burnIt.hbs: ```handlebars{% raw %} {{#setAsUser (getUsernameFromParams url=../urlParams)}} {{#setActivityStyleFromUser user}} <h1>Burn those {{activityStyle}}s, {{user.name}}!</h1> {{/setActivityStyleFromUser}} {{/setAsUser}} {% endraw %}``` Hmm... let's try logging `activityStyle` just above `{% raw %}{{#setActivityStyleFromUser}}{% endraw %}`. So the code will now look like this: ```handlebars{% raw %} {{#setAsUser (getUsernameFromParams url=../urlParams)}} {{log activityStyle}} {{#setActivityStyleFromUser user}} <h1>Burn those {{activityStyle}}s, {{user.name}}!</h1> {{/setActivityStyleFromUser}} {{/setAsUser}} {% endraw %}``` No change! We still get the same error message on the same line. ### Time to call a friend We could try rooting around like this forever -- but let's not. DuckDuckGo to the rescure: ![Several DuckDuckGo search results -- none look promising](/img/2015/ddg-search-results.png) Ugh, too many results and after clicking through several of them, there's nothing useful. Looking directly on stackoverflow [doesn't yield anything helpful](http://stackoverflow.com/questions/25555793/typeerror-cannot-read-property-typekey-of-undefined) here either. ### Why is everything so fked up? This question is an opportunity. It's a chance to take stock of what we've just gone through and ask ourselves why. Why is hunting this bug so hard? There's barely any code written in this project, and already: + The templates look like a transcript of the delusional ramblings of a crazy person + The error messages are really long but tell us nothing + The control flow is untraceable Can we figure it out? Maybe. But if we keep writing code this way, someday it will paralyze our ability to fix bugs or develop new features, and in my book that's harmful. ## The Truth about Handlebars The core of [handlebars is written in Bison/YACC](https://github.com/wycats/handlebars.js/blob/master/src/handlebars.yy), a metalanguage used to create new programming languages like PHP, Ruby, Go, Bash, PostgreSQL. Handlebars itself is a very minimal Domain Specific Language which is why you saw "compiler.js" and "runtime.js" listed in the stacktrace at the top of this post. While implemented in javascript, code written in handlebars is running in its own compiler and runtime, on top of javascript. Handlebars is extensible through the use of helpers that you register with the compiler. Think of helpers not like functions in javascript, but more like macros in C++. You're literally adding capabilities to the compiler. So, as you write additional helpers for Handlebars, you're _effectively_ extending Handlebars into your own DSL. Surprise! In my stupid source code used to write this post, you can start to see this happening in the helpers I created: ```javascript{% raw %} Handlebars.registerHelper('eq', function(a, b, options) { if (a == b) { return options.fn(this); } else { return options.inverse(this); } }); Handlebars.registerHelper('set', function(key, value) { this[key] = value; return ''; }); {% endraw %}``` I've re-implemented conditionals and variable assignment operators! Does that sound like the purview of templates to you? It doesn't to me. ## But I'm not just picking on Handlebars I've seen senior Python developers do exactly the same kind of thing using [Jinja2](http://jinja.pocoo.org/) or even Django templates. If I had done any Ruby on Rails development, I'm quite sure I would find the same defects there as well. PHP is perhaps the worst offender in this regard because it was explicitly developed to be embedded into html between `<?php` and `?>` tags. WordPress templates in particular suffer from the comingling of application logic and presentational markup leading to total spaghetti code. At least the debugging tools for PHP were built with this comingling in mind. Except that PHP now has [Twig templates](http://twig.sensiolabs.org/) which are Handlebars for PHP... What all these templating languages have in common is that they split your application logic across multiple contexts, typically without your awareness. The fancier the templating language, the more the temptation to build your application in a blindspot that you can't easily debug. It's like putting your diamonds in the one corner of the jewelry store where there aren't any cameras. ## There must be a better way There is. Write your application logic in javascript (or whatever your main application language happens to be). Stacktraces will tell you something useful, other people will help you on stackoverflow, and more importantly the your business logic will be expressed in a language that's used by thousands of developers. But don't throw Handlebars away just yet -- we have to have some way to put data into HTML, and Handlebars (despite my grumblings here) can still be a good tool for that purpose. Here's my ten commandments for safely using Handlebars: ### 1. Thou shalt print static values Never compute anything in a template. Pour only finished data into the template. Lean on `{% raw %}{{var}}{% endraw %}` expressions as much as possible to display values. ### 2. Thou shalt not mutate variables As data passes between different scopes, the same data should retain the same variable name and the same path. You should never do this: ```handlebars{% raw %} {{firstName}} {{#myHelper who=firstName}} {{who}} {{/myHelper}} {% endraw %}``` When variables change names inside handlebars, it becomes exponentially harder to trace them from child scopes to parent scopes. This leads to misunderstandings of how the code works and bugs over time. When these bugs arise, they're very costly to fix because of the effort of tracking them down. ### 3. Thou shalt (cautiously) use `#if` and `#each` Collections of data are ubiquitous. Handlebars already has a built-in blocker helper for `{% raw %}{{#each}}{% endraw %}` that iterates over an array to produce html. Using `{% raw %}{{#each}}{% endraw %}` as an iterator is an excellent idea. It's documented, has test coverage, and you can find support for it on stackoverflow/irc/etc. Checking for the presence of a variable can be an acceptable way to structure a template. It's okay to check for a user in the template context before showing account information, for instance. ### 4. Thou shalt not `switch` Checking for the presence of a variable is completely different than executing an extended else-if conditional/pattern matching in templates. Your template is too dynamic -- move more logic back into the server. ### 5. Thou shalt not `else` I'm gonna be stickler on this one. Your code _will_ be clearer without this extra logic. You're building templates, not lambda calculus. Steer clear of any unnecessary conditional logic. ### 6. Thou shalt be LOOSE The pythonistas have focused on the programming discipline called DRY. The key insight is that application logic works better with a single source of truth. Code that is duplicated often needs to be updated in sync, but can easily get out of sync. Template logic is governed instead by the need to be loosely coupled. Overly dynamic templates are difficult to reason about because of the many codepaths they are tied to. It's generally much easier to maintain multiple similar templates with isolated responsibilities than a single template with many responsibilities. When these similar template get of sync, they are easy to change because there are few side effects. ### 7. Thou shalt not use more than seven mustaches in a row I just made up the number seven, but seriously though too many mustaches next to each other are unreadable. If you have that many mustaches consider moving more logic into client or server side javascript. ### 8. Thou shalt the use of minize partials Partials are there to DRY up your code, which is a bad thing. Sometimes, you will absolutely have to reuse something, like the footer, across different templates. Consider ways to avoid creating a partial if possible, but don't feel too bad doing it. The real danger is that you'll start passing very different contexts to the partial. By keeping as much of the template in a single file as possible, you avoid switching contexts and the possibility of mutating a variable. ### 9. Thou shalt program declaratively One of the reasons that the sample code above is _so_ bad is the gratuitous use of procedural logic in the template. If you must use helpers, only use them declaratively, as shown in this Handlebars documentation snippet: ```handlebars{% raw %} {{{link "See more..." href=story.url class="story"}}} {% endraw %}``` If there's verb used in the handlebars helper you did it wrong**. ### 10. Thou shalt create an API for client side rendering If you want templates that can be rendered client side and server side, move all the data generation, templation selection, and other logic into the server and expose an api endpoint for your client side code to consume. This keeps your program logic DRY (as it should be) but keeps logic out of the templates (where it shouldn't be). ## In Conclusion Handlebars isn't evil, but it can be a siren song luring you to dive overboard into a sea of unrelenting bugs. By keeping your programming logic in your main programming language (javascript in this example), you can keep your code easier to debug, easier to read, and easier to find help when you get stuck. When you do use Handlebars, keep it stupidly simple. You'll thank me later. </code></pre> How to Kill Email - Why startups will always fail to displace email 2014-11-21T00:00:00+00:00 It's painful to watch startup after startup brand themselves the "email killer." This post is the real story of how to kill email. Make It Backwards Compatible</h2> Email is so well entrenched in both business and personal communications that it cannot simply be "killed" outright. According to Experian</a> Marketing Services (a scary treasure trove of voting records, email marketing, credit history, rental history, etc.), the volume of email traffic year over year continues to grow</a> by double digits. Radicati predicts continued double digit growth in email volume</a> over the next four years. In the face of growing email volume, whatever replaces email must also be backwards compatible with it. This is how git largely replaced svn -- developers could keep all their existing commit history when making the switch. Similarly, HTML5 was able to easily replace HTML4 because it mostly added tags and APIs on top of HTML4. Perhaps supporting POP3 (which was not built for the smartphone era) should be optional, but whatever kills email must support IMAP and SMTP. </a> Create a New Email Standard</h2> In a word, we need standards not apps to kill email. This is why so-called email killers such as Slack and Yammer are failing to dislodge email. What Do You Mean, "Email Standard"?</h3> Roughly, I mean a protocol such as SMTP + IMAP. We need a clearly defined technical specification that has been hashed out by the Internet Enginerring TaskForce (IETF) or some other standards body with representatives from spam filtering companies, networking experts, systems administrators, government agencies, and email client vendors. Why Consensus Is So Important</h3> Without consensus, broad adoption is impossible. HTML5, CSS3, and ES5 (to name a few) could not have come about without a consensus between browser makers and web developers. Today, it's tough to find a website made in the last 3 years that doesn't use some aspects of HTML5 or CSS3. To take a parallel case from the online video technology industry (currently mine), MPEG-DASH has relatively strong consensus between video transcoding and CDN companies, but it seems that web developers were not given enough voice in the process. Ask any web developer about their experience with MPEG-DASH, and you'll hear about the horrors of trying to parse XML in the browser, among other complaints. For this reason, MPEG-DASH hasn't gotten much traction at this point in November 2014. Other Reasons Why Apps Won't Kill Email</h3> Battle-tested ideas not subject to whims of a few charismatic founders</li> Zero cost experimentation allowing a critical mass of technical innovation</li> Stability of APIs</li> Resilience to disappearing funding or acquisition</li> Competition from a variety of implementations</li> Protection against one company selling ALL user data</li> </ul> Each of these could almost justify it's own blog post -- if you're curious about these, the comments at the bottom are the perfect place for a flame war. Game on! </a> Make It Irrestible</h2> The benefits to this new email standard must be compelling and obvious. Without bells and whistles for all parties from sysadmins to email client vendors to end users, there's no way to get traction. If you've read this far, you know that my opinions alone are woefully inadequate for a standard. I imagine that whatever replaces email must at least provide the following features (nearly all of them relate to the problem of bad actors). Feature: Impostors, Begone!</h3> Phishing attacks are devastating and nearly unpreventable using the current email protocols. In particular, high profile targets like CEOs are often the victims of carefully researched and planned phishing attacks. This has to stop. A Russian hacker must not be able to convince my Grandma that she is me. Feature: Want to Spam? Make it Rain</h3> Sending lots and lots of physical mail is expensive. Email should also be computationally expensive to send (perhaps using something analogous to mining for bitcoins?). I believe that shifting costs onto spammers would solve the spam epidemic we face today. Caveat: Never Miss Your Friends Updates</h4> The danger here is that if sending emails is too "expensive", no one will use it. Spam mitigation must take into account the needs for legitimate notifications. For instance, Facebook really does need to send emails almost every time a user posts a comment. Perhaps some mechanism of trust (modeled on keybase.io or perhaps on the block chain) could be used to allow trusted, verified senders' messages to always be delivered. Admittedly, this is a hard problem, and it will require the experise of Facebook and serious cryptologists. Feature: Keep Your Viruses to Yourself</h3> Malware transmitted via email is a huge problem. If we could "sanitize" email so that viruses could not be transmitted via attachments, I would force my brother-in-law to sign up in a heart beat. Feature: Make The Conversation Stick Together</h3> Let's use a UUID to bind together messages on a thread. We could eliminate resending the entire thread, and make all clients present a time ordered sequence of emails grouped by the thread UUID. Feature: Tie Message Format to HTML5/CSS3</h3> The bad actor here is Microsoft Outlook (and every web based mail client like Gmail and Yahoo!). It's still like 1995 for marketers and nonprofits sending pretty emails. It sucks and it has to end. Concluding Thoughts</h2> Postal mail is more sophiscated than email these days. That's a shame. It's time for the tech community to stop talking about killing email and actually do it. However, we can't do it alone. We need a shiny new standard with bells and whistels that's broadly agreed upon, widely adopted, and backwards compatible. That's how to kill email. Securing SSH with Multiple Keys 2014-09-04T00:00:00+00:00 It's a great idea to use specific ssh-key key pairs per service (or per repository even). Even if your keys are compromised, they don't allow your attacker to access any other services. Unfortunately, these services often assume you'll use the default key pair (id_rsa). 1. Get Organized</h2> I like to make folders inside ~/.ssh for each service (Github, AWS, BitBucket, DigitalOcean, etc), and place various keypairs. EX: ~/.ssh/ | - id_rsa | - id_rsa.pub | | - github/ | | - workuser | | - workuser.pub | | - other | | - other.pub | | - bitbucket/ | - personaluser | - personaluser.pub </code></pre></div> To actually create the key pairs, use ssh-keygen</code> in the terminal. 2. Upload your public key</h2> All of these services have a settings page for adding SSH public keys. Add yours, and also include a descriptive label if you can. Usually, I describe which computer this public key came from so that I can revoke with confidence later on, if necessary. 3. Find the Git Push endpoint</h2> For bitbucket and github, the endpoint is typically in this format: git@github.com:brycefisher/defaulterrors.git</code> Let's break that down: git -- everything before '@' is the username for SSH</li> github.com -- the actual hostname</li> brycefisher/defaulterrors.git -- path to the git repository</li> </ul> 4. Config all the things!</h2> Open up ~/.ssh/config in your favorite text editor. At the end add a new entry (separated by a new line above): Host github Hostname github.com User git IdentityFile ~/.ssh/github/brycefisher </code></pre> Choose whatever you want for Host</code> -- that's an alias you can use in ssh. Notice how all the pieces from 3. above mapped to this new entry in ~/.ssh/config. In particular, we used everything except the path to the git repo. 5. Add a git remote</h2> Under the hood, git is using SSH, and it can access the alias we created earlier (github) to reach github. Our git remote will have to know the SSH alias and the path to the git repo. $ git remote add origin deferr:brycefisherfleig/defaulterrors.git </code></pre>6. Make Sure it Works</h2> $ git push -u origin master </code></pre> If the above command succeeds, you'll see something like this: Counting objects: 3, done. Writing objects: 100% (3/3), 262 bytes | 0 bytes/s, done. Total 3 (delta 0), reused 0 (delta 0) To github:brycefisherfleig/defaulterrors.git </code></pre> If instead you see: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. </code></pre> Then re-check the steps above. 7 Alternatives to Amazon CloudFront CDN 2014-05-27T00:00:00+00:00 AWS CloudFront has many bizarre quirks: only two invalidations of 1000 paths can run concurrently</li> each invalidation takes about 5-15 minutes to complete</li> there is no "clear all cache"</li> configuring gzip compression is nontrivial</li> </ul> I'm comparing the alternatives to CloudFront for static web content (html, css, js, images), specifically Azure</a>, CDNify</a>, CloudFlare</a>, CDNSun</a>, Fastly</a>, KeyCDN</a>, and MaxCDN</a> . I've picked only CDNs that made pricing information available (which eliminated most of the options I could find). Pricing is a feature, and typically CDNs without pricing information didn't disclose their feature set online either. One CDN that looked fantastic but didn't make the cut was Highwinds</a>. They provide loads of documentation and a free trial, but no pricing information. Pricing Comparison</h2> CDN</th> Pricing Model</th> Costs (US)</th> HTTPS Pricing</th> Free Option</th></tr></thead> CloudFront</td> Per GB</td> $0.12</td> $0 - SNI</td> Y - 1 yr</td></tr> Azure</td> Per GB</td> $0.12</td> Unknown</td> Y</td></tr> CDNify</td> Flat Rate + Over 150GB</td> $10.00 + $0.07</td> $0</td> N</td></tr> CloudFlare</td> Flat Rate</td> $20.00</td> $200/mo Plan</td> Y</td></tr> CDNSun</td> Per GB</td> $0.45</td> $699/yr</td> Y - 15 days</td></tr> Fastly</td> Per Request + Per GB</td> $0.01 + $0.12</td> $500 setup + $100/mon</td> Y - 30 days</td></tr> KeyCDN</td> Per GB</td> $0.04</td> $0</td> N</td></tr> MaxCDN</td> Flat Rate + Over 100GB</td> $9.00 + $0.08</td> $39/mo</td> N</td></tr> </tbody></table> "Per GB" (normal font) refers to bandwidth charges</li> "Flat Rate" (in bold) is per month fees. "Over ___ GB" specifies the included bandwidth</li> "Per Request" (in italics) refers to individual requests between edge and client</li> </ul> The standard going rate for CDN bandwith is 12¢. KeyCDN is the cheapest option, and CDNSun is priciest. Network Comparison</h2> CDN</th> HTTPS</th> POPs</th> Latency</th> Gzip</th> Minification</th></tr></thead> CloudFront</td> Y</td> 50</td> 47</td> Y - in origin</td> N</td></tr> Azure</td> ?</td> 30</td> 47</td> Y - in origin</td> N</td></tr> CDNify</td> Y</td> 20</td> 77</td> Y</td> N</td></tr> CloudFlare</td> Y</td> 25</td> 49</td> Y</td> Y</td></tr> CDNSun</td> Y</td> 85</td> ?</td> ?</td> Y</td></tr> Fastly</td> Y</td> 20</td> 58</td> Y</td> N</td></tr> KeyCDN</td> Y</td> 15</td> ?</td> Y</td> Y - images only</td></tr> MaxCDN</td> Y</td> 15</td> 44</td> Y - in origin</td> Y</td></tr> </tbody></table> There is no information about using HTTPS with Azure in the Microsoft documentation or pricing</li> "POPs" or Points of Presence refers to how many geographically separate data centers a CDN has (number of servers per data center can vary)</li> Latency is measured in milliseconds by Citrix ITM</a> (formerly "Cedexis") as of 5/26/2014</li> A "Y" for Gzip means that the CDN can transparently handle compression during transmission over the network for you. "Y - in origin" means your origin server must handle compression, but the CDN can intelligently serve compressed files</li> </ul> The only big difference between these CDNs IMHO is whether or not the CDN can handle Gzip compression for you without configuring an origin server. Ideally, you could just dump your pre-minified files onto a cloud storage service such as AWS S3 or RackSpace CloudFiles and let the CDN handle this for. Caching Behavior Comparison</h2> CDN</th> Headers</th> Cookies</th> Query String</th> Protocol</th></tr></thead> CloudFront</td> Y</td> Y</td> Y</td> N</td></tr> Azure</td> Y</td> Y</td> Y</td> ?</td></tr> CDNify</td> ?</td> ?</td> ?</td> ?</td></tr> CloudFlare</td> Y</td> N</td> Y</td> Y</td></tr> CDNSun</td> Y</td> Y</td> Y</td> ?</td></tr> Fastly</td> Y</td> Y</td> Y</td> Y</td></tr> KeyCDN</td> Y</td> ?</td> Y</td> ?</td></tr> MaxCDN</td> Y</td> Y</td> Y</td> ?</td></tr> </tbody></table> "Headers" refers to respecting the origin's Cache-Control (or other caching specific) headers</li> All columns refer to the ability to vary the cached content served to visitors by examining this information</li> </ul> The winner in this category appears to be Fastly, with CDNify providing virtually no information about how configurable its caching behavior is. Configuration Comparison</h2> CDN</th> Latency</th> Purge All</th> API</th></tr></thead> CloudFront</td> 10 mins</td> N</td> Y</td></tr> Azure</td> mins?</td> ?</td> N</td></tr> CDNify</td> mins?</td> Y</td> Y</td></tr> CloudFlare</td> secs</td> Y</td> Y</td></tr> CDNSun</td> mins?</td> Y</td> Y</td></tr> Fastly</td> 150 ms</td> Y</td> Y</td></tr> KeyCDN</td> 5 mins</td> Y</td> ?</td></tr> MaxCDN</td> 30 sec</td> Y</td> Y</td></tr> </tbody></table> "latency" means the time required to update the edge servers with configuration changes (or purges/invalidations)</li> Except for AWS CloudFront, all latency information is taken from the self-reporting of each company's website</li> "mins?" means that the company did not state how long, and I assume changes will take minutes to propagate across edge servers</li> </ul> Fastly comes out ahead in this category, with MaxCDN and CloudFlare close behind. And the Winner Is...</h2> For me, it's a toss up between KeyCDN and MaxCDN. I really like the fast configuration updates that MaxCDN boasts, but I wish the SSL/TLS pricing model was a bit cheaper. If you just want to get a CDN out there and don't want to do much configuration, Key CDN seems like the way to go. Fastly is my third place runner up. The incredible documentation, low latency, and extreme configurability make it the power house in the group. Sadly, hosting a custom SSL certificate on Fastly costs a small fortune. Using AWS CloudFront Behaviors with HTTPS for Planet Drupal RSS 2014-04-30T00:00:00+00:00 After all the relevations of Snowden last year, I've been wanting to be more security focused. A small but obvious change is providing this blog over HTTPS. Disqus powers the comments, and if you login via Disqus I'd like to protect my visitors session cookies or authentication information. I felt so proud [to have setup a CDN powered, encrypted blog]({{ site.url }}{% post_url 2014-04-24-setting-up-ssl-on-aws-cloudfront-and-s3 %}). Unfortunately, I discovered that RSS feed aggregators</a> and validators are incompatible with HTTPS</a>. Based on my experience, it seems that the Drupal.org aggregators that power the illustrious Planet Drupal also fail in the face of HTTPS. Oh, Behave - a CloudFront workaround with Behaviors</h2> If you're using CloudFront, it's actually very simple to serve your feeds over HTTP and redirect everything else to HTTPS. Let's assume you already direct all your traffic to HTTPS. Here's how to add an exception: Open CloudFront in Web Console</li> Edit the Distribution</li> Click on "Behaviors"</li> Add a new behavior, setting the following settings: Path Pattern - setup a regex that matches your RSS feeds. I set all my RSS feeds as feed.xml, so for me this value was "/feed.xml"</li> Origin - choose your existing Origin from the dropdown list</li> Viewer Protocol Policy - use the default setting here (this setting is the important one)</li> </ul> </li> Save it!</li> </ol> Once your distribution finishes updating the edge servers, you should be able to access your feed over HTTP (and HTTPS), but everything else should redirect to HTTPS. Tada! Why Would RSS Aggregators Not Be HTTPS Compatible?</h2> Dave Winer, one of the creators of feedvalidator.org, writes that switching the aggregator to HTTPS has a huge potential downside of losing parts of your audience with software that doesn't support HTTPS. Furthermore, he writes: ... All aggregators will support HTTPS if enough developers of feeds require it. If it should turn out that way, ... I'll do it. And I'll be pissed because it's time I'd rather spend doing something creative. ... I can't see for the life of me why you need to push RSS over a secure connection. :-) </blockquote> Dave has a point</h3> It does seem unnecessary, actually, to encrypt connections between RSS feeds and aggregators. If the feed is meant to be public, then hopefully no sensitive information is being transmitted in the feed. Also, I hate the drudgery of meaningless work and would be loathe to impose it on anyone else. Maybe we should do it anyway.</h3> Since December 2012, the new HSTS HTTP headers</a> have taken hold in browser-land. These headers allow servers to instruct browsers to only access the server over HTTPS for a certain period of time. It's like a cache TTL. HSTS headers are designed to prevent certain kinds of man-in-the-middle attacks. If a website is following the best practice of using HSTS headers in response to the pervasive threat that the NSA, GCHQ, China, Russia, and roving bands of hackers, then that site won't be able to syndicate a feed from the HSTS domain. Finally, what about private RSS feeds? Shouldn't we be able to leverage run of the mill aggregators for use cases such as for UPS package delivery? What about notification from the government that our new passport has been approved and will be shipped Monday? I can also imagine private feeds from Etrade notifying customers that an automated transaction occurred. The possibilities for private feeds are really endless, especially once one starts considering the coming Internet of Things. Avoiding the Duplicate Content Penalty with AWS S3 and CloudFront 2014-04-26T00:00:00+00:00 As soon as I setup this blog on CloudFront using S3 as the origin server, my friends immediately started telling me I was at risk of the duplicate content penalty</a> Google imposes on sites that have nearly identical content. After many hours of pouring over Stackoverflow to no avail, I found two strategies to protect my blog: canonical urls and user-agent based S3 bucket policies. If you're hosting your blog or site on S3 without CloudFront, the duplicate content penalty is not an issue for you in the same way. Don't attempt to use the S3 bucket policies described here. Strategy 1: User Agent Policies in AWS S3</h2> This strategy actively shields your content from being discoverable by search engine crawlers, so it's likely to be much more effective. Unfortunately, it's not fool proof and it's dependent on AWS CloudFront implementation details that could change without any public notice. In particular, we're relying on the User-Agent string that CloudFront uses for it's GET requests to your S3 bucket. Here's what to do: Go your AWS Web Console and go to the S3 service</li> Expand the Permissions tab for your website's bucket</li> Click "Edit Bucket Policy"</li> The JSON snippet in the Policy Editor should look like this:</li> </ol> { "Version":"2012-10-17", "Statement": [ { "Sid":"PublicReadGetObject", "Effect":"Allow", "Principal": { "AWS": "" }, "Action":"s3:GetObject", "Resource":"arn:aws:s3:::bryce-fisher-fleig-org/" } ] } </code></pre> Underneath the line starting with "Resource", add this:</li> </ol> "Condition": { "StringEqualsIgnoreCase": { "aws:UserAgent": "Amazon CloudFront" } } </code></pre> The whole should probably look something like this: { "Version": "2012-10-17", "Statement": [ { "Sid": "Allow CloudFront to read from Bucket", "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::bryce-fisher-fleig-org/", "Condition": { "StringEqualsIgnoreCase": { "aws:UserAgent": "Amazon CloudFront" } } } ] } </code></pre>Did It Work??</h3> If you did everything right, your site should still be accessible through the CloudFront url. Check a url that hasn't been cached in CloudFront yet (or do an invalidation). Go check right now, I'll wait. If new urls are coming through CloudFront okay, then make sure that your S3 website endpoint returns 403 Access Denied. Go, look up the website endpoint in your S3 console. If you received some kind of 403 error, Google's crawler shouldn't be able to access your S3 bucket directly. Congrats! No more duplicate content penalty for you. Safety First</h3> Obviously, you'll need to change bryce-fisher-fleig-org to the name of your bucket. Also, you'll need to make sure that whatever other permissions you have make sense with this rule, which is beyond the scope of this article. I recommend setting up S3 to log into a separate bucket so that you can see the access log of what CloudFront is doing to your website bucket. Why CloudFront Origin Access Identity Sucks</h3> As a quick aside, CloudFront does provide a mechanism that allows direct access between CloudFront and S3 without all this user agent smoke and mirrors nonsense. It's called Origin Access Identity, but it's incompatible with S3 when you enable the static website hosting mode for your bucket. If AWS gets around to fixing Origin Access Identity, you won't need this hack anymore. Fingers crossed! Strategy 2: Canonical Url Links</h2> Let's take a look at the second strategy to avoid duplicate content penalties: canonical urls in a link tag. All you have to do is a link tag with rel canoncial and href http://your/real/url/here</code>. You can look at the source code of this page, or do something like the following: <link rel="canonical" href="https://bryce.fisher-fleig.org/blog/avoiding-duplicate-content-penalty-with-aws-s3-and-cloudfront"> </code></pre> This technique is much weaker in that we're providing suggestions to Google's crawlers, but we can't force the crawlers to comply. Nevertheless, if your mother always taught you it's not polite to user agent sniff, canonical url links are completely kosher. Also, they won't randomly take your site down when CloudFront gets a new user agent. This technique can be really handy on Drupal sites (which provide most content at two urls) or potentially for marketing campaigns with various url parameters or in other situations</a>. Setting Up SSL on AWS CloudFront and S3 2014-04-24T00:00:00+00:00 I setup this blog using S3 as the origin server, CloudFront as my CDN, and HTTPS for roughly $10. Since there were so many articles to read along the way, I've gathered up what worked for me here. Step 0: Background & Assumptions</h2> CloudFront offers HTTPS at no additional charge in certain situations. CloudFront takes advantage of an extension to the TLS protocol called Server Name Indication ("SNI"), which allows servers on a single IPv4 address to serve multiple domains over HTTPS. It's a huge cost savings for Amazon, and they've decided to pass on the savings to you. Pay attention to step 5 below to take advantage of SNI. You'll still have to buy your own SSL certificate. I'll assume you're already using Jekyll or another static site generator to create HTML files that you plan to host on S3. Throughout this tutorial I'll use the domain name of this blog to help you understand. You'll want to replace "bryce.fisher-fleig.org" with your domain name anywhere you see it below. Step 1: Create an S3 Bucket</h2> Bucket Name: Your bucket name must be like bryce-fisher-fleig-org to work with CloudFront (bryce.fisher-fleig.org won't work). Permissions: Grant everyone read access to this bucket so that CloudFront can read out the content. Here's how: Open the Change the Permissions tab on your shiney new bucket</li> Click "Edit bucket policy"</li> Copy and paste this JSON snippet into the policy editor:</li> </ol> { "Version":"2012-10-17", "Statement":[{ "Sid":"PublicReadGetObject", "Effect":"Allow", "Principal": { "AWS": "" }, "Action":["s3:GetObject"], "Resource":["arn:aws:s3:::bryce-fisher-fleig-org/"] }] } </code></pre> Change "bryce-fisher-fleig-org" above to your bucket name</li> Click Save</li> </ol> Static Website Hosting Open this tab and select "Enable website hosting". Make sure to set your Index document to index.html so that folders work right. Keep the "Endpoint" url handy for later. This would be a great time to start uploading all the things to your new bucket, and try clicking on the Endpoint url to make sure things are groovy. Step 2: Create a CloudFront Distribution</h2> Choose "Web"</li> Origin Domain Name - Paste the "Endpoint" from S3's Static Website Hosting tab here</li> Minimum TTL - Click "Customize". Keep this at 5 minutes (360 seconds). Once everything is groovy, change it to something higher (for instance, 24 hours).</li> Price Class - I use US, Europe, and Asia. Choose whatever you're comfortable paying for.</li> Alternate Domain Names - Enter "bryce.fisher-fleig.org" (or whatever your subdomain is). You'll have to use Route53 if you want to use a root domain</a> with CloudFront.</li> Default Root Object enter "index.html". Otherwise, http://bryce.fisher-fleig.org/ wouldn't display the root index.html file.</li> </ol> Don't worry about the various HTTPS settings yet, we'll come back to that later. None of the other settings need to be changed either. Step 3: Obtain an SSL Certificate</h2> This step walks you through paying a company called a Certificate Authority ("CA") to verify your identity and issue an SSL certificate with their name on it. No matter which CA you pick, you'll have to create a Certificate Signing Request (CSR) which you will upload to the CA. You'll also need to generate a matching file called a private key. The CA's act as a kind of notary public for the internet. Anyone can generate an SSL certificate for free, but if your certificate isn't signed by an official CA, then Chrome, Firefox, Safari, etc will not allow visitors to access your site over HTTPS. There's a limited number of these Certificate Authorities around, thus they can and do charge obscene prices. Resellers, like the two I mention below, offer much better prices. Different CA's offer different kinds of certificates with various degrees of verification and encryption. For a personal blog, choose a single domain certificate verified using Domain Validation ("DV"). If you're doing e-commerce, you probably want the more complicated and expensive process called Extended Validation ("EV"). Step 3.A: Buy an SSL Certificate on the Cheap</h3> The best place I can find legit certs is ClickSSL.com</a>. They provide an awesome search filter, and their certificates are all signed using SHA-2 (more on this below). All certificates have 2048-bit key length meaning that your certificate has reasonable encryption strength for the near future. Previously, I advocated using CheapestSSLs.com, but I've discovered that they can have slow customer service and weaker encryption. It's unclear if they support SHA-2. CheapestSSLs.com is a little cheaper than ClickSSL.com. With either reseller, my main criteria for a certificate are: SHA-2 (sometimes also called "SHA-256") -- Chrome and Firefox have pledged to start showing really scary security warnings to users for sites using the older SHA-1 signing algorithm by 2017. They already print warnings in the console today.</li> Domain Validation (aka "DV") -- this means you can get the certificate right away through a self-service process. This option is only available for single domain certificates.</li> Price</li> </ol> At my renewal time, I chose the RapidSSL certificate for this domain because I only want to serve one domain using this certificate, and the price was only $27 for two years. I highly recommend buying a two year certificate so that you don't have to go through this process every single year. Step 3.B: Generate a matching private key and CSR</h3> Once you've paid for a certificate, generate a private key and a certificate signing request (CSR) using this command in the terminal: $ openssl req \ -sha256 \ -new -newkey \ rsa:2048 -nodes \ -keyout bryce_fisher-fleig_org.key \ -out bryce_fisher-fleig_org.csr </code></pre> (There's a great article on what this command is doing at entrust.net</a>, but don't worry too much about it.) Openssl will ask you a series of questions once you enter the command above: Country Name - two letter ISO 3166-1 code for your country</a> (e.g. "US")</li> State or Province - e.g. "California"</li> Locality Name - e.g. "San Francisco"</li> Organization Name - your DBA, LLC, corp, or personal name. E.g. "Bryce Fisher-Fleig"</li> Organizational Unit - optional, type "." to leave blank</li> Common Name - This is the one you MUST get right. Use the domain name you want to serve over HTTPS. E.g. "bryce.fisher-fleig.org"</li> Email address - This is the email that will be exposed to spammers on the public certificate. You must have it, but consider creating a dedicated email address here for spam to collect in. E.g. "bff.dns.spam@gmail.com"</li> </ol> Later on, we'll provide this private key to CloudFront, but it's vital that you keep the private key very, very private. The security of your website depends on the private key being accessible only to you and CloudFront. You should never email or share this file with anyone for any reason, and you shouldn't store this file Dropbox or Google Drive. If the key is ever compromised, many CA's will allow you regenerate your certificate for free. Step 3.C: Upload the CSR to your Certificate Authority</h3> This part was very simple for me with both Comodo and RapidSSL. I did have to re-enter some details that I entered on the CSR such as the domain name. Step 3.D: Get validated!</h3> Honestly, verifying that I owned the domain is the worst part every time I've done this. I can't complain about price or customer service, but the process seemed to take forever. Thank yourself for choosing Domain Validation now, because you've saved yourself a week of effort. There's three ways to complete the DV process: Click a link sent to a special email address (usually administrator@fisher-fleig.org). This seems the hardest to me, because you have to setup the email server ahead of time, add MX records, create email accounts, and possibly setup anti-spam measures. I try never to do this, but this is only option for RapidSSL. If you have to go this route, I highly recommend using gandi.net</a>. They provide all the email setup for you as part of all domain name purchases, and their prices are very reasonable.</li> Host a file at a special url on that domain. This is tricky, but usually easier than email. Comodo supports this option. If you use S3, just upload the file they give you to S3.</li> Set a dns TXT Record. This is easiest possible way. Unfortunately, I've never used a CA that supported this option. If you find one, please tell me in the comments!</li> </ol> If you don't hear anything right away, you failed the domain validation. Don't wait! Contact customer support immediately to help you resolve the issue. I've found support helpful, but the response time was unbearably slow (24hrs or more). Step 4: Upload the Certificate to IAM</h2> Amazon stores all of the SSL certificates used by any of the AWS services inside it's Identity Access and Management (IAM) service. In order to upload an SSL certificate you must create an IAM user with sufficient permissions. The root account user will not work. Create an admin IAM user:</h3> On the AWS Web Console, go to the IAM service</li> Click on the "Users" tab</li> Click "Create New Users"</li> Enter the first user name. E.g. "SslKingPin"</li> Click "Create"</li> Show or download the credentials -- we'll need them a little bit later.</li> Click on the newly created user</li> In the user console at the bottom of the page, choose the "Permissions" tab</li> Click "Attach User Policy"</li> Copy and paste this JSON snippet into the Policy Editor and click save:</li> </ol> { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [""], "Resource": [""] }] } </code></pre> The JSON snippet above gives this user all privileges, so be careful! Setup AWS Cli</h3> For Mac just use homebrew: $ brew up $ brew install aws-cfn-tools awscli </code></pre> On Debian/Ubuntu, just apt-get: $ sudo apt-get install python python-pip $ sudo pip install awscli </code></pre> For other platforms, see the official AWS documentation</a> Configure the AWS Cli:</h3> $ aws configure </code></pre> The only tricky part here is to make sure you enter the IAM credentials for our SslKingPin user. The other settings are a matter of preference. Finally! Upload the Damn Certificate:</h3> Download the zip file from Comodo containing your new SSL certificate.</li> Unzip all the files into the same directory you have the CSR and private key files.</li> Follow these steps to upload your ssl certificate to IAM:</li> </ol> $ cd /path/to/certificate/files/ $ aws iam upload-server-certificate \ --server-certificate-name bryce_fisher-fleig_org \ --certificate-body file://bryce_fisher-fleig_org.crt \ --private-key file://bryce_fisher-fleig_org.key \ --certificate-chain file://PostivieSSLCA2.crt \ --path /cloudfront/ </code></pre> This command is REALLY tricky to get right. So, lets break it down a bit: server-certificate-name This is just an alpha-numeric label you see inside AWS consoles. It doesn't matter to anyone except you.</li> certificate-body The certificate from Comodo</li> private-key This is the very first crypto file we made in Step 3.B.</li> certificate-chain Using Comodo, this turned out to be the PositiveSSLCA2.crt file. It's basically any files between but not including the trusted root certificate and the certificate with your domain name on it.</li> path This ends in a slash, and it must start with /cloudfront/.</li> </ul> Step 5: Configure CloudFront to use SSL</h2> Inside the CloudFront dashboard, edit the distribution you created in step 2.</li> On the General tab, click "edit"</li> SSL Certificate - choose "Custom SSL Certificate (stored in AWS IAM)" and choose the certificate name from step 4 in the drop down (e.g. "BryceFisherFleigOrg").</li> Custom SSL Client Support - choose "Only Clients that Support Server Name Indication (SNI)". This saves you tons and tons of money, but means older versions of IE and Android can't use the SSL.</li> Click "Yes, Edit"</li> Click the "Behaviors" tab</li> Edit the behavior there.</li> Viewer Protocol Policy - choose "redirect HTTP to HTTPS" to force everyone to use SSL when viewing your site.</li> Click "Yes, Edit"</li> Go "back to distributions". You'll see that the status is "In Progress" until the change takes effect all over the world.</li> </ol> Step 6: Point your DNS to CloudFront</h2> From the CloudFront distribution screen, copy the "Domain Name" value. It will be something like zxy9qwnududu7.cloudfront.net. Go to your DNS nameservers and create a CNAME for this value. These changes will take some time to propagate fully (for me about 10-15 minutes, but your mileage may vary). Conclusion</h2> Did you find an error in my tutorial? Did something not make sense? Tweet or comment below and I'd love to help you figure this out. Happy blogging! A Super Handy Translation Checklist 2014-03-14T00:00:00+00:00 I've been bitten time after time during (what I wrongly assumed) would be the last phase of a project because of translation issues. So, I've put together a checklist of some DO's and DON'Ts that I've learned that help me review someone else's code or even my own code before forking over the big bucks to a translation company. This checklist will be most helpful if you use Drupal as your CMS, and you send your translation vendor XLIFF and POT files. Also, this checklist assumes you have i18n already setup on your Drupal instance, and you're looking to deploy a new feature or set of content. Enjoy! Drupal/PHP</h2> Proper use of t() All text wrapped in t()</li> All t() strings contain a coherent thought for translators to work with rule of thumb, make sure each t() is fed at least 3 words</li> No t() invocations wrapped within another t()</li> Include FontAwesome or other image font tags into t() where it is a semantic part of the text content. (For instance, in Korean, it might be correct to place the glyph in the middle of a string, or at the beginning, whereas in English it must come last)</li> </ul> </li> Avoid line breaks in t() All text must flow naturally. Line breaking is something you must let go of (embrace the Zen)</li> </ul> </li> </ol> </li> All links to localized content will localize (ex: lookup the nid of node translations and use url() or l() )</li> For multibyte text that needs to be used by JS, render text inside tags with explicit IDs</li> All public facing pages are attached to nodes (ajax urls are an exception)</li> No public facing pages use hook_menu</li> </ol> JS</h2> Avoid language specific paths in Ajax requests wherever possible</li> Avoid passing strings around in JS when working with multibyte strings (Asian characters)</li> Do use JS polyfills to support CSS powered buttons, etc in old browsers (progressive enhancement)</li> </ol> CSS</h2> All elements containing visible text can expand 2X vertically (or horizontally) without breaking the layout</li> Background images on all expandable boxes gracefully stretch or contract (good use of background-size</code> or gradients)</li> No side-by-side floated divs suddenly wrap to next line (unexpectedly) when content grows or shrinks</li> Avoid narrow columns (less than 300px) as much as possible</li> No textual string content inside pseudo-element content</code> rules -- use t() instead</li> Use HTC polyfills (css3pie, etc) if your project requires you to support old versions of IE</li> </ol> IMGs</h2> Check that NO images referenced in HTML have text</li> Checked that NO background images referenced in CSS contain text</li> Avoid image based buttons (use CSS instead wherever possible)</li> Avoid image based gradients (use CSS wherever possible)</li> </ol> Architecture</h2> All public facing pages are use nodes to exploit url aliases (except for ajax urls)</li> No public facing pages use hook_menu</li> </ol> Communication</h2> Prepared a list of nodes that need aliases and (re)translation for translation team</li> Prepared a list of tokens (ex: !token, @token, %token) that must NOT be translated to give to translation team</li> Prepared a list of other quirks that the translation team needs to know about</li> Prepared some "next steps" for the translation team once they've finished </li> </ol> PHP: What Does function ...() use () syntax mean? 2014-01-31T00:00:00+00:00 I've recently seen an unfamiliar syntax all over the place, but I didn't understand it and I couldn't find anything about it while searching. Here's a sample: <?php // ... $loop->onReadable($server, function ($server) use ($loop) { // ... }); ?> </code></pre> From igorw's github</a>. Did you catch it? Why is there a "use" statement after a function? Googling "php use function" wasn't any help. Buried Deep in the PHP Manual - A Clue</h2> After an hour of wandering through the PHP documentation (which I actually quite like doing anyway), I discovered this very helpful comment</a>: The keyword 'use' has two different applications, but the reserved word table links to here. It can apply to namespace constucts: </blockquote> Namespaces are pretty cool. They form the foundation of PSR-0, Composer, and everything good in the PHP these days. But, that syntax is used at the top of files on it's own line. Then, I found the answer: The 'use' keyword also applies to closure constructs: </blockquote> Closures: A (JavaScripty) World Within</h2> I've seen the term "closure" in PHP before, but I'm most familiar with it inside Javascript. A closure in PHP simply refers to any anonymous function, even in the global scope. However, PHP 5.3 introduced this sneaky new syntax whereby you can copy variables out of the parent scope and into the "closure" scope. Nifty, huh? Here's what the PHP documentation has to say on the use syntax</a>: Closures may also inherit variables from the parent scope. Any such variables must be passed to the use</code> language construct. Inheriting variables from the parent scope is not the same as using global variables. Global variables exist in the global scope, which is the same no matter what function is executing. The parent scope of a closure is the function in which the closure was declared (not necessarily the function it was called from). </blockquote> Parameters by Any Other Name</h2> Just like normal function parameters, parameters provided to the closure scope via the use</code> keyword are passed by value. To pass parameters by reference, simply add an ampersand (&) in front of the parameter. Exciting Uses of Closures and the Use Keyword</h2> I've been trying to figure out to hack together a pure PHP server using PHP 5.3 for a special side project, and I came upon the excellent static site generator called Sculpin</a>. It's PHAR can be called from the command line just like Jekyll. It also has a special flag --server</code> that spins up a server on the spot, even using PHP 5.3! Intrigued, I dug into the source code and finally found an evented PHP server inspired by NodeJS</a>. Here's a few key parts of the code: <?php // ... $httpServer = new ReactHttpServer($socketServer); $httpServer->on("request", function($request, $response) use ($repository, $docroot, $output) { $path = $docroot.'/'.ltrim(rawurldecode($request->getPath()), '/'); if (is_dir($path)) { $path .= '/index.html'; } if (!file_exists($path)) { HttpServer::logRequest($output, 404, $request); $response->writeHead(404); return $response->end(); } // ... $response->writeHead(200, array( "Content-Type" => $type, )); $response->end(file_get_contents($path)); }); $socketServer->listen($port, '0.0.0.0'); // ... </code></pre>A Brighter Future</h2> I really hope that the future of PHP is a future that's less tied to Apache. That worked well in the past, but it's just too hard to provision and tweak Apache when you just want to write. Source code like this inspires me to believe that it's possible to handle all of our routing logic without having to rely on some other piece of software to ferry requests from the network to our application and back again. Where have you seen use</code> done right? Let me know your thoughts on this and other esoterica from recent versions of PHP. Using hook_menu to capture custom menus in code 2014-01-14T00:00:00+00:00 Every time I use hook_menu()</code>, I get an awful headache. Why? Drupal has overloaded this hook to handle both the menu system and the routing system, which seems like an architectural mistake to me. As a result, hook_menu()</code> has many confusing and poorly documented features in the very lengthy api documentation</a>. I'm going to walk you through some rather simple steps to using hook_menu()</code> to create public-facing menu items. I won't get into the complexities of the routing system. Why Would We Want to Use This?</h2> Since it's so easy to create menu items inside the admin pages, why would you ever want to create a custom menu item in code? Any time you want to create a new path that links to custom functionality, you'll want to reach for hook_menu()</code>. Drupal does a great job of handling url aliases to nodes and other entities (taxonomy, users, etc), but the admin forms don't allow you to trigger custom PHP code at an arbitrary path. An additional benefit that I'll touch on briefly is access control. This hook allows you to specify what permissions a given path will require of the user. So, any time you need really custom roles or permissions, consider using hook_permission() (aka hook_perm in Drupal 6) to create a new permission. Then read the api documentation above, specifically the "access arguments" section. I'll leave this as an exercise for the reader. Starting with the Basics</h2> This hook will only be detected by Drupal inside of a module. You can't put it in template.php. Also, it's important to know how to trigger this function, since Drupal only runs it rarely. In Drupal 6, you'll have to visit the modules page. In Drupal 7, it seems to be enough to clear all caches (you may have to run cron as well). This is important because it's to be fooled into thinking that your changes to hook_menu were incorrectly done. One commenter suggested a technique for automatically refreshing hook_menu during development.</a> I should also note that it's good to look at the admin menu page periodically to make sure that your changes worked. In drupal 7, you can find the list of menus at 'admin/structure/menu'. In Drupal 6, check out 'admin/build/menu'. Getting our hands dirty hook_menu()</code></h2> Let's talk about the function signature. Don't accept any arguments into your function. Do return an array, which is by convention named $items</code>. The outline of the function will look like this: <?php function mymodule_menu() { $items = array(); return $items; } </code></pre> This is correct, but won't do anything yet. Creating Menu Items</h3> To actually create some menu items, we'll need to set the paths for the menu items as keys in the $items</code> array. Here's a simple example: <?php function mymodule_menu() { $items = array(); $items['status'] = array(); return $items; } </code></pre> The key line above is $items['status'] = array()</code>. The array key 'status' tells Drupal to listen for requests at that path or route. (I use the terms "path" and "route" interchangeably throughout this article.) We'll need to fill in some more code to tell Drupal how to respond to requests on that route: <?php function mymodule_menu() { $items = array(); $items['status'] = array( 'title' => 'My Status Page', 'menu_name' => 'main-menu', 'page callback' => 'mymodule_status_page', 'access arguments' => array('access content'), ); return $items; } </code></pre> The nested array takes several different keys. This is where the online documentation gets really hard to read. Some keys are incompatible or nonsensical with other keys, and other keys are required but only in certain situations. The above code is one combination of keys that will work to create a new top level menu item in the Main Menu labelled 'My Status Page'. Here's a detailed explanation: title - This key allows you to set the text shown to the user on the menu</li> menu_name - Just what it sounds like. If you omit this, Drupal defaults to the Navigation menu which is only shown to authenticated users. Set this key to whatever menu you want the item to appear in.</li> access arguments - This array is passed to the function user_access()</code> to check if the current user has all those permissions. I always use access content</code> so that if the site content needs to be put behind a paywall, or otherwise hidden later on, it will default to whatever the rest of the site content is set to.</li> </ul> Page Callback Functions</h3> The 'page callback'</code> specifies a callback function name. So, I would need to write a function mymodule_status_page(). If your custom menu item is outputting html, you'll want to eventually output using the theme('page', ...)</code> function. For example: <?php function mymodule_status_page() { // Do custom stuff here $html = ''; //Put the final markup in $html // theme('page') will wrap your output with a // page in the create theme (admin or public). print theme('page', $html); } </code></pre>Hierarchy: Taking it to the next level</h2> What if we wanted to nest a menu item? There are two ways of doing this. 1. Automatic Nesting Based on Route</h3> By default, Drupal will look at the registered paths in the menu/router system, and it will organize them hierarchically. So, if we wanted to add a subpage underneath our 'status' route (defined above), we could create a new route at 'status/us-east-1'. Here's the code we'd need to use to do that: <?php function mymodule_menu() { $items = array(); // "Parent" route $items['status'] = array( // add the keys here ); // "Child" route $items['status/us-east-1'] = array( // add keys here ); return $items; } </code></pre> I've omitted the keys ('title', etc) for clarity's sake, but all the keys on each route work just as we discussed above. I appreciate the fact that adding routes in this manner will work seamlessly with custom menu items created on menu admin form. Neat! For more examples of nesting using this method, I do recommend checking out the api documentation</a>. 2. Explicitly Specifying a Parent Menu Item</h3> What if you wanted to make 'status/us-east-1' the parent of 'status' in the menu? Unfortunately, there's no safe way to do that across browsers other than manually dragging the menu items around in the admin form. The reason we can't do that safely is that we don't a reliable to figure out what the menu id (called mlid</code>) is in the database until after hook_menu()</code> runs. This is a really unfortunate problem with the existing system that lumps menu and router systems together. Fortunately, we can rearrange menu items by specifying the parent menu id (called the plid</code>) for known items. The plid</code> number will almost certainly be specific to the database you're working on, so I wouldn't use it for a contrib module. However, this is perfectly acceptable for your own freelance or agency custom module development. Here's how it works:</h3> Create the child menu item in $items</code></li> Find the parent menu item we want to attach the child to</li> Find the 'mlid' value of the parent menu item</li> Add a new key to the child menu item called 'plid' and set the value you found in step 3.</li> </ol> The tricky step here is finding the mlid value of the parent. The simplest way to do that is to use the admin form page and mouse over the parent menu item edit link. Use that number at the end of url: Here's what that code looks like: <?php function mymodule_menu() { $items = array(); // "Child" route $items['status'] = array( // add the other keys here 'plid' => 218, // mlid of the parent ); return $items; } </code></pre> Considerations for Multisite Drupal 2013-12-28T00:00:00+00:00 We've been using the Domain Access the module</a> at my day job for about 5 years with Drupal 6. Recently, we've decided Domain Access is not the right solution for our use case. Domain Access</h2> Strengths</h3> According to the module's project page, Domain Access (emphasis mine) provides tools for running a group of affiliated sites from one Drupal installation and a single shared database. The module allows you to share users, content, and configurations. </blockquote> The problem this module aims to solve is allowing multiple domains to have identical content. If you're trying to build multiple sites with substantial amounts of identical content, definitely use the Domain Access module. It's fantastic at making the very same information available on different subdomains (one.example.com and two.example.com) and different top level domains as well (one.com and two.com). One of the best features of Domain Access is the ability to share users and passwords across different sites using the same Drupal authentication system you already know. Awesome! Pitfalls</h3> Tight Coupling of Dependencies If you're using Domain Access, the same modules are powering your site. This could be a good thing or a bad thing depending on your perspective. From my point of view, this is more of a problem than a solution. For example, imagine a new version of Views comes out and you have twenty views with heavy customization on each domain of 5 domains. You have to be sure that upgrading the Views module won't break 100 different views before you deploy the new module. Personally, I'd rather test only 20 views at a time rather than 100. Any big change to any of the sites (especially contrib modules) means you should really be checking the admin forms and a good sampling of urls and functionality across each domain. It sounds like less work to maintain only one Drupal instance that powers multiple websites. In my experience, it's almost always more work to force one Drupal instance to work across a spectrum of different sites, each with different requirements and stakeholders. It ends up being less work to ship any given piece of code when each site is it's own island. SEO Snafus In the six months I've been at my current job, I've gotten numerous complaints of nodes being indexed at the wrong domain. Domain Access only checks for permission of the current user for a given path. This means that you can't have two different nodes at the same path at different domains. Each node must have a unique path across all domains. Additionally, if a user mistakenly accesses a node that exists at a different subdomain, they will get a 403 error (access denied) instead of 404 error (not found). Each site builder must be trained on how to use set the appropriate the domain access settings. If nodes are configured improperly, Google can (and will!) index the wrong domain for that node. Many times, certain content types are only properly themed for one domain, and when they are indexed at a different domain, the content looks horrible. Views and Node Access Unfortunately, Views will mostly respect the user access settings of a given node. At least in Drupal 6 and probably Drupal 7, this means that you can't easily create views with teasers of content from another domain unless that content is accessible on the same domain as the view. For instance, at our main corporate site we have a blog teaser. Unfortunately, we couldn't create a view because each of our blog posts was only allowed on our blog domain. As a workaround, we've created RSS feeds using views on the blog site, then ingested that feed on the corporate site in many instances. For other situations, we've abandoned Views and written SQL directly. SEO Penalties Additionally, when using this module correctly, major search engines will sometimes penalize urls that showing identical content</a> at different domains. Use Separate Drupal Databases</h2> Another way to approach a multisite situation is to create new database on a MySQL server, and create another settings.php</code> file. The advantage of this technique is that you can share module code across different Drupal instances, but you'll avoid all the issues with duplicate content showing up on the wrong domains. This is a huge win, and could be a much better solution for your multisite needs. However, you'll still run into all the "tightly coupled dependencies" issues that arise whenever you need to upgrade modules or Drupal core. The biggest problem with this approach to multisite Drupal is that you can't share users and passwords between different Drupal instances. This is actually a nontrivial problem — you don't want to have different usernames and passwords across affiliated sites. Just imagine if you had a separate account for every stack exchange affiliate! There are several solutions to this problem, but they all involve various trade offs. I plan to discuss this issue more in a future post. Don't Do Multisite</h2> This is my preferred solution. Each domain that is (or could reasonably become) different should have it's own fully independent Drupal instance, code and data. This approach means that you don't have to review every site after every change, and you won't have problems with content being indexed on the wrong domain. I would guess that for most organizations that want to use multiple domains or subdomains, they do want unique content on unique domains. (It's actually hard to imagine a situation where you'd want the same content on different domains). You also avoid the tightly coupled code issues that arise for the separate settings.php file. The "downside" is that you'll have to upgrade each site individually. However, this is feature not a bug in my mind. As I mentioned before, I'd love to do less testing and manual QA for each code deployment. I don't mind spending a little extra time testing my changes in exchange for greater confidence that my changes won't break a totally different site. Best Practices for Translation in Drupal 2013-12-13T00:00:00+00:00 One of the biggest reasons my company decided to use Drupal for their marketing website</a> is the capacity to translate and "localize" content. Drupal has a series of APIs going back at least to Drupal 4.6 that make it possible to have different paths for various pieces content. Drupal also has the ability to import and export translations of strings stored in code as well as content stored in nodes, taxonomy terms, and more. Most translation vendors can accommodate the exported content into their software and ship back a "localized" version of that content. Recently, as I've been involved in this process, I've discovered several pitfalls and how best to avoid them. I'll be focusing this post on Drupal 6 since that's what I use most often, but I'll also try to mention some things about Drupal 7 as we go. I won't tell you how to get started localizing your website</a> -- that's a whole book in and of itself. These are more guidelines for how to minimize friction between themers, site builders, and translators. Block Level Elements and the t() function</h2> The t() function</a> takes English strings that are used in PHP and exposes them to Drupal for translation. Drupal takes care of the hard part of figuring out which language to display on the current page, but Drupal won't know to translate your string unless you've first wrapped it in the t(). The API documentation for t()</a> has lots of great examples of correct and incorrect usage of t(). The bottom line is inline elements should go inside t() and block level elements outside.  <p><?php print t('Your new BFF in web development'); ?></p>  <?php print t('Your new BFF in web development'); ?> </code></pre> Notice that paragraph tags are block level elements in CSS, and therefore should not be included in t(). However, inline elements like a</code>, strong</code>, em</code>, and br</code> should definitely go inside t(). NOTE: There's lots of ways to get translation strings out of Drupal and expose English text for translation inside Drupal. You only need to use t()</code> for English text inside PHP code. You don't need it inside nodes (for instance). More on this later. Escaping Values Inside t()</h2> Often times, you'll want to pass a value that shouldn't be translated into t()</code>, such as a URL, the number of users currently signed in, etc. However, it's really, really important NOT to get fancy. Each use of t()</code> creates a separate string for a translator to figure out. You'll often have to make judgment calls about how to break up the strings into meaningful bits. Always, step back and ask yourself, "What does the translator need to see to understand this text the way the use will see it?" Example 1: Don't Use Unnecessary Plurals</h3> Again here's some examples: <?php // Correct print t( 'Providing !visitors visitors with outrageously awesome Drupal advice', array('!visitors' => $num_visitors) ); // Yikes! Confusing print t( 'Providing !visitors visitors with outrageously awesome Drupal advice', array('!visitors' => format_plural($num_visitors, '1 visitor', '@count visitors')) ); ?> </code></pre> Let's unpack that a bit. You can tell t()</code> to insert variables into your strings using a special escape syntax. Use ! or @ or % to tell Drupal where to insert your variable. Then pass t()</code> an array of values. So, the above examples are just inserting the value of $num_visitors</code> into each string above. Technically speaking, the above "confusing" example is not wrong. However, the translator won't see the plural translation strings ('1 visitor' and '@count visitors') strings alongside of the main string. The first string is better because it keeps the whole sentence in one single string that the translator can figure out much more easily. There may be situations where you could one or multiple numbers inside t()</code> variables. It's perfectly appropriate to use the format_plural</code> function in those situations, but you may want to give your translation vendor a heads up about that string. I've seen translators consistently unable to figure out this sort of problem — and I don't blame them. It's our responsibility to make sure translators eceive content they can understand. Going the extra for your translators will save you time and money by not having to send back strings for re-translation over and over again. Example 2: Don't Use Abstraction</h3> I've seen some programmers try to separate repeated strings into different t()</code> invokations. Let's look at an example HTML output in English:  <ul> <li>Bryce Fisher-Fleig - Astounding Drupal Insights</li> <li>Bryce Fisher-Fleig - Jaw Dropping Coding Skillz</li> <li>Bryce Fisher-Fleig - Oh, snap! He did it again</li> </ul> </code></pre> We're working in PHP and we see a pattern, so why not DRY up the code a little bit? So, trying to use "abstraction" on these strings might lead to something like this:  <ul> <li><?php print t('Bryce Fisher-Fleig - !description', array('!description' => t('Astounding Drupal Insights'))); ?></li> <li><?php print t('Bryce Fisher-Fleig - !description', array('!description' => t('Jaw Dropping Coding Skillz'))); ?></li> <li><?php print t('Bryce Fisher-Fleig - !description', array('!description' => t('Oh, snap! He did it again'))); ?></li> </ul> </code></pre> This is an antipattern because the translator might need to reorder these strings, or they might translate the text different having known the strings appear next to each other. Here's an example of how that might need to appear in Japanese:  <ul> <li>驚異Drupalの洞察 - Bryce Fisher-Fleig</li> <li>顎落としたりするスキルコーディング - Bryce Fisher-Fleig</li> <li>ああ、スナップ！彼は再びそれをやった - Bryce Fisher-Fleig</li> </ul> </code></pre> If you break the string into pieces, translators won't be able to see that the strings belong together and you'll have wasted your money on the translation effort. Always keep strings together for the translator just like you would for the end user. I promise you the translators will be thoroughly confused if you try to separate the repeated strings. Let the experts make the call on whether or not a given phrase should be translated the same way in different contexts. Just keep the strings together as much as possible so the translators can have at least full sentence context, like so:  <ul> <li><?php print t('BryceAdamFisher - Astounding Drupal Insights'); ?></li> <li><?php print t('BryceAdamFisher - Jaw Dropping Coding Skillz'); ?></li> <li><?php print t('BryceAdamFisher - Oh, snap! He did it again'); ?></li> </ul> </code></pre>Preparing to Export Strings from t()</code></h2> If you do use escaped values inside t()</code>, you'll need to notify the translators that words starting with ! @ or % need to left as is because they will be replaced by strings automatically inside Drupal. Each time you start translating fresh content, it might be wise to notify your vendor just in case they've forgotten or changed some settings in their translation software. I'd also highly recommend grooming through your templates for fancy usage of t()</code> that could be simplified or made clearer. Doing so will save you time and money, and make the whole process smoother. Other Ways to Make Strings Translatable</h2> In Drupal 6, using the the i18n module</a> and related modules allows you to translate blocks, taxonomy terms, node titles, node url aliases (which is awesome!), and node body fields. However, there's no good way to translate custom CCK fields. Menus are easier to simply create multiple versions of that are only visible for a specific language. In Drupal 7, you can also leverage the entity translation module</a> to translate custom fields. Content Lifecycle</h2> One final best practice I want to make special note of is never manually edit translation files. Your project manager wants to ship code (and so do you!) but if you playing God with your Korean text, bad things will happen. Never never ever edit the import or export files directly. These files are designed to be used in conjunction with automatic processes so you don't have to mess with translation. That's also why you're paying good money to language experts. So why would you screw it up at the last minute? Do your due diligence to prepare your templates by properly using the t()</code> in clear, unambiguous ways, and stay out of the import/export files. You'll thank yourself. Quick Drupal - Lessons Learned While Whipping Up a Fresh Drupal Instance 2013-11-29T00:00:00+00:00 I've recently migrated to Ubuntu 12.04 from Windows, and I ran into some issues getting the drush core-quick-drupal command. As a quick refresher, the "quick drupal" command will download, install, and even login to a new drupal instance on your local machine. It's really hand for debugging and also contributing patches back to drupal.org because you eliminate any side effects from bizarre customizations. Coming from a Windows background there were quite a few hurdles I had to figure out -- so I thought I'd record my experiences here for other poor souls trying to use quick drupal on Ubuntu. Some of these tips will doubtless be helpful on Mac or Windows as well. Problem - Old Drush Installed</h2> Ubuntu uses a package management tool called "apt-get" to install, update, and remove applications from the operating systems. If you've ever used composer for PHP, or npm for NodeJS, it's exactly the same, and it's my favorite feature of linuxy operating systems over Mac and Windows. Conveniently, apt-get has a package for drush! $ sudo apt-get install drush </code></pre> Unfortunately, the apt-get version of Drush is version 4.x, whereas the stable version of Drush is 5.x. Also, the quick drupal command we're insterested in is only added to drush starting somewhere in the 5.x branch. However, drush has it's own self-update command. Hooray! $ drush self-update -y </code></pre> Phew! Now if you run drush help</code> in the terminal, you'll see that the command "core-quick-drupal" listed near the top. Problem - RTFM</h2> I thought I was an experienced drush ninja, but it turns out I hadn't fully read the documentation carefully for core-quick-drupal command. One reason I love the quick drupal command is that it doesn't require you to setup a mysql database (you can use SQLite which is much simpler for testing patches) and you don't need to configure Apache (more on that later). You just type this one command and everything magically works. To use quick drupal as I had intended, you actually need to use the first optional parameter. So, the command will look like this: $ drush core-quick-drupal patches -y </code></pre> Notice "patches" in the command above. "patches" is the site name parameter, and you must specify this parameter or else drush will try to use mysql for datastore, and if it fails you'll have all kinds of weird errors. The moral of the story here is to always "read the fucking manual" as the old adage goes. Problem - SQLite Not Installed</h2> Eventually, I realized that SQLite does not come pre-installed on Ubuntu 12.04 (at least, not in the Desktop distro that I downloaded). I also found some blogs that mentioned PHP5 requires sqlite v3 to work and the php5 sqlite connector. I use apt-get to install these dependencies: $ sudo apt-get install sqlite3 $ sudo apt-get install php5-sqlite3 </code></pre> If you try the quick drupal command again, you should start to get helpful error messages at this point. Problem - Required PHP Extensions Missing</h2> If you've followed along with me so far, you can probably figure the rest of this out on your own just fine. However, this next hurdle blocked me for a while because the error message in the terminal had html around it. With my particular setup, I had use apt-get to install php on my machine but certain extensions required by Drupal were not installed as part of apt-get command. In my case, I found that the "gd" graphics library was not installed. Fortunately, a compatible version of gd for php5 can be easily installed: $ sudo apt-get install php5-gd </code></pre>Use a Compatible PHP Binary</h2> The quick drupal command was intended to be used with PHP 5.4 because this version of PHP has an embedded web server that obviates the need for Apache or Nginx (or whatever). It's very easy to get PHP 5.4 with a full WAMPP stack on Windows using Apache Friends' XAMPP installer. I highly recommend that approach since you won't have to follow any of the steps in this tutorial to get up and running with Drupal. Ubuntu's apt-get only has PHP 5.3 which doesn't have the php web server built-in. Furthermore, drush core-quick-drupal doesn't like the cli flavor of PHP 5.3 either. So, unless you want to compile PHP and all the necessary extensions from source, I recommend you install the PHP cgi binary: $ sudo apt-get install php-cgi </code></pre>Emulating the Web Server on PHP 5.3</h3> If you try to run quick drupal now, Drush has a fantastic error message that tells you exactly what to do. You'll need to download a library from github that runs a pure php httpserver</a>. This is only necessary for php 5.3. You'll need to unpack this tarball in the /usr/share/drush/lib</code> directory. When you're done, make sure that this file exists: /usr/share/drush/lib/youngj-httpserver-41dd2b7/httpserver.php</code> Sadly, quick drupal still won't work without one more component. Setup A Local Email Server</h2> Quick Drupal is starting to not be so quick, you're probably thinking. But, it will be once you get this command working. You're almost there! The only thing missing at this point is a working email server. Drupal needs to be send out an administrator email when you start the quick drupal instance. There's probably a way around this step, but it's actually easier than I had thought. Dummy Mail for Windows</h3> If you're running Windows, just use SMTP 4 Dev a dummy SMTP server</a>. It's so easy... you'll thank yourself a millions times for using this program. A "Real" Email Server for Ubuntu</h3> On Ubuntu, we're going to install postfix and mailutils, and then we'll configure them. There's a great stackoverflow that explains postfix setup nicely</a>, but I'll explain how I did it slightly differently. $ sudo apt-get install postfix </code></pre> During the install screen in the terminal, I choose "local server" (the last option). This means that postfix will only deliver emails locally to the current machine. The next screen will ask you for a Fully Qualified Doman Name. This doesn't matter -- I entered localhost and it worked fine. Checking Email in the Terminal</h3> Next you'll need to install the command line mail client: $ sudo apt-get install mailutils </code></pre> Try sending yourself a test email now like this: $ mail -s "Testing" root@localhost Cc: [ENTER] [CTRL-D] </code></pre> Pressing CTRL + D ends the message body and sends the message. You should be able to see that message was sent (Null message body; hope that's ok</code>). You should be good to go! Tell PHP How to Use Postfix to Email You the Admin Password</h3> Then under /etc/php5/conf.d create a file (e.g. mailconfig.ini) with these contents (note you'll need sudo permissions to save the file in this directory): sendmail_from = "root@localhost" sendmail_path = "/usr/sbin/sendmail -t -i -f root@localhost" </code></pre>It Works!</h2> Finally, we can now run the quick drupal command...quickly. Here's how to do it: $ cd /path/to/where/drupal/should/be $ drush qd my_drupal -y </code></pre> A few minutes later, you'll have a php server running a Drupal instance on SQLite, and your browser will open and login you in as the admin user. Hooray! Now go make some patches. Getting to Done Faster with Drush and PHP 2013-11-22T00:00:00+00:00 If you're like me, you've been using Drush ("DRUsh SHell") for a while to make several tasks less tedious: downloading and enabling modules, running database updates, backing up the database, clearing the drupal cache, running drupal cron, and resetting the admin password. Drush does all those things really well for drupal 6 and 7. It just works! This post is about using Drush as part of your coding process for theming or module development (or even core contributions). Drush and PHP have several commands that I've come to find invaluable while I write code. These commands let you run small snippets without waiting for a whole page to load in the browser. Simply pop open your shell, and start typing. This can be invaluable to make sure that the code behaves how you think it will. This makes your code faster to write and less prone to error because you'll already know what the code is doing better. One final argument for using these techniques is isolation. It's often hard to trigger exactly the piece of code that you want to understand. All these commands allow you focus on a specific piece of code without the distractions and complications of a whole site. Command 1: Evaluating PHP snippets in Drupal drush ev</code></h2> This command evaluates a single line of PHP code in the context of drupal. I use this all the time to test that various Drupal API functions work as I expect. Here's a quick example. Let's say you're not sure if the l()</code> function will return the URL or the anchor tag inside your template function. Here's what do: $ cd /path/to/drupal $ drush ev "print l('node/1');" http://example.com/node </code></pre> Great! We can now see how this particular function will work without doing a bunch of other stuff that takes 10-15 seconds. We now know exactly what to put inside our template file. Here's another simple example of drush ev</code>. Let's say we've written a helper function in our custom module that takes a nid and returns a field value. However, you wrote this function last month you're sure if the return value is an object or an array or a string. Let's find out: $ cd /path/to/drupal $ drush ev "var_dump(my_module_helper(1));" array(1) { 0 => "hello drush" } </code></pre> You've just saved yourself several minutes digging through your own code. Well done! There's a million ways to use this code. I often use drush ev</code> to test each command I'm interesting in while implementing a function for the first time. This prevents me from running into surprises 99% of the time and has made me much more productive. Caveats with drush ev</code></h3> Drush won't print anything to the shell when the command ends, so you'll need to use print_r()</code>, print</code>, echo</code>, or var_dump()</code> to see the result of the command in the shell. The big limitation of drush ev</code> is that you can only execute one liners. If the code you want to execute can't be called in one line, see if the next technique is more appropriate. Command 2: Interactive php shell with php -a</code></h2> This command opens an interactive php shell which is really useful for prototyping a sequence of built-in php commands that can't be fit into a one-liner. This is awesome for experimenting with array operations, reflections in PHP, regex functions like preg_match()</code>, or other confusing syntax. Any time you're not sure how a php function will work, give this a shot. Mac and Linux Examples of interactive PHP Shell</h3> On OSX and Linux, here's how this works: $ php -a interactive shell php > $x = serialize(array(1 => 'abc')); php > echo $x; a:1:{i:1;s:3:"abc";} </code></pre> Notice that you still need to issue some sort of print/var_dump command to see the output. However, you can create and manipulate as many variables as you want on as many lines of code as you want. Windows Example:</h3> I no longer have a windows machine at home, but this works a little differently on Windows. Here's an example from memory: C:\> php -e <?php $x = serialize(array(1 => 'abc')); echo $x; ?>CTRL-Z a:1:{i:1;s:3:"abc";} C:\> </code></pre> So, the main thing to note is that php on windows will treat whatever you type as a regular php file which requires the use of to work. Also, php won't start interpreting until you press CTRL + Z and ENTER after the closing ?>. So, this is less useful on windows, but still really handy when you need it. Command 3: Executing SQL Queries with drush sqlq</code></h2> If you've ever found yourself typing SQL on the command through the mysql client, it can be a real pain to look up the user name and password just to execute some simple queries. Drush has your back! It uses it's magic to execute sql queries in the context of the Drupal database with drush sqlq</code>. This command is really useful for figuring out sql querys that change the database. For instance, if you want to write a module update that unpublishes all nodes of a certain type, you could try out the query using this command. To see the effects, however you'll still need to look at the live site directly or use a MySQL gui client like Sequel for Mac or Heidi SQL for Windows. A Word about Best Practices</h2> The drush commands I've shown you are really powerful. And, as uncle Ben taught us in Spiderman, "with great power comes great responsibility." Please do not use these commands on a production website. These commands are awesome at debugging your problems locally, but they can really mess up your website. They are not "sandboxed" in anyway. Okay, with that said, go out there and write some great code that works the first time you run it! Settings.php Hack - Keep Credentials out of Your Code 2013-11-12T00:00:00+00:00 Over my three year Drupal career, I've seen several interesting server setups for running Drupal. One technique I've been gravitating toward is storing the environment specific configuration settings in the environment. ##Motivation For This Hack As the web moves more and more into cloud computing, we developers need to get comfortable letting our code run in a new and unforeseen range of contexts. I believe that building our software to depend on environment variables is an excellent way to build more portable code that is flexible enough to thrive in these changing conditions. Currently as I write this post in November, 2013, I've been running this site on shared hosting through HostGator — not the most exciting choice these days, but it is an affordable option. On many shared hosts including HostGator, databases must be prefixed by your username. However, I like to run my databases using different credentials locally. Also, to setup a staging environment on the same server, you'd need to create a separate database with separate credentials. My final argument for this particular technique is transparency. I store the code for several side projects on Github as open source projects, but I really don't want the credentials stored there for all to see. By removing credentials from the code, I feel much more confident about using Github this way. Conveniently, Github does not charge for storing open source projects (but they do charge for hosting private repositories). ##Playing with Environment Variables The trick to getting credentials out of your settings.php file is to use environment variables. The shells for Windows, Mac, and Linux are able to store small pieces of information in the environment. It's really easy to temporarily set an environmental variable in a bash shell: $ set x='y' $ echo $x y </code></pre> Here's how it looks in a Windows command prompt: > set x='y' > echo %x% 'y' </code></pre> However, these environment variables will poof out of existence as soon as your restart your computer (or close that terminal session!). In order for Drupal to work, we need a "set it and forget it" approach to these environment variables, so that they persistent indefinitely. Long Lived Environment Variables in Windows</h3> </iframe> In Windows Vista, Windows 7, or WIndows 8, you can set a persistent environment variable by right clicking on "My Computer" > "Properties", then clicking on "Advanced system settings" and clicking "Environment Variables" at the bottom of the dialog box. Then click "New..." and you'll be able to name and set your variable. This variable will be available system wide. Restart Apache and you'll be good to go! If you're following along on Windows, try setting an environment variable called MY_ENVVAR and set the value to helloWeb. Then skip down to the section on "Testing" Persistent Environment Variables for Apache on Mac and Linux</h3> There are at least two different ways to accomplish this. If you're more experienced in this department, please chime in the comments and I'll update this section of the post. 1) Root Acces, Baby!</h4> If you do have root access to your Mac/Linux machine, then you should be able to edit the file /etc/apache2/envvars. You can set an environment variable for Apache inside this file like this: export MY_ENVVAR="helloWeb" </code></pre> Using Ubuntu, I had to restart the machine before MY_ENVAR was picked up by Apache. I also needed to restart apache after making a change: $ sudo apachectl restart </code></pre>2) Using Htaccess</h4> If you don't have root access, never fear! You can still set an environment variable for Apache inside an htaccess file. Create or edit a file named .htaccess in the directory above your public root, and add the following code to it: SetEnv MY_ENVVAR "helloWeb" </code></pre> For this to work, you may need to have IT set Apache's AllowOverride directive to "All" or "FileInfo" for your virtual host. Try it out first, then contact support at your hosting company if this doesn't work (see the testing section next). Testing Your New Environment Variable</h3> No matter how you set it, we need to make sure that php picks up the environment variable you set. This is super easy. Just a make a file called envvars.php and put it the top directory of your drupal site. Put this inside your file: <?php var_dump(getenv('MY_ENVVAR')); ?> </code></pre> We'll use the getenv()</code> function</a> to retrieve the environment variable. During debugging, I use var_dump()</code> so that I can see what type of variable php is returning. Now go to your drupal site's url and add '/envvars.php' to the end. If you see: string(8) "helloWeb" </code></pre> It worked! You're ready to go on to the rest of this article. If you see: bool(0) FALSE </code></pre> Then, php didn't pick up this variable. First try restarting Apache on that machine. If that doesn't give you the expected result, then try contacting your hosting company's support for assistance or leave me a comment on this post. Using an Environment Variable in Settings.php on Drupal 6</h2> Drupal 6 uses a single connection string to store the database credentials. This string contains the user name, the database name and password, the host, and port. Here's an example: mysql://user:password@localhost/database </code></pre> Simply replace 'user' with your database username, and so on. Then set an environment variable with this string for Drupal 6 and the php function getenv</code></a> to retrieve the value. Easy! Setting and Retrieving Database Credentials on Drupal 7</h2> Drupal 7 uses an array to store the database credentials. Although you could store separate parts of the credentials in separate environment variables, I prefer to use the serialize()</code> function</a> to convert an array into a string ahead of time. I typically do this by opening an interactive php shell and copying the result: $ php -a Interactive shell php > $creds = array(); php > $creds['driver'] = 'mysql'; php > $creds['database'] = 'databasename'; php > $creds['username'] = 'username'; php > $creds['password'] = 'password'; php > $creds['host'] = 'localhost'; php > $creds['prefix'] = ''; php > print serialize($creds); a:5:{s:6:"driver";s:5:"mysql";s:8:"database";s:12:"databasename";s:8:"password";s:8:"password";s:4:"host";s:9:"localhost";s:6:"prefix";s:0:"";} </code></pre> You'll want to replace all the dummy values above with your actual database credentials, and then copy and paste that last line exactly as it appears into an environment variable. Inside my settings.php, I use the following code to retrieve the environment variable: <?php // Store the database credentials serialized in an environmental variable. $db_creds = unserialize(getenv('DRUPAL_DB_CRED')); $databases = array ( 'default' => array ( 'default' => $db_creds, ), ); </code></pre>Potential Pitfalls</h2> One downside of this approach is that when you set up Drupal on a new environment, you'll have to remember what all the environment variables are that need to be set, and you'll need to set them up appropriately. One way of addressing this problem is to make sure that your project is structured so that the directory above your webroot (public_html, html, www or whatever) is also in source control. Then make a text file documenting these environmental variables above the public root. I would probably call this file INSTALL.TXT since that's a fairly accurate description of the information you want to convey others and future you. If you're using github, create a wiki page to document this information. Alternative Means to the Same End</h2> An alternative approach that sometimes works well in practice is to create a single file that contains all the different variables that change based on environment, and then add in detection logic inside that file to set variables appropriately. While this is a really transparent, centralized way to store all this information for a small number of environments, it quickly gets out of hand for any large number of environments or contexts that you want to account for. Theoretically, this also adds some overhead to each page request that Drupal handles as well (probably even Drupal native cache hits), although this is probably neglible in practice. You shouldn't be mixing code and configuration any more than you have to. Configuration (like database credentials) should be changeable without forcing your code to be rewritten. By separating the code and configuration as I've done above, you have an elegant scalable solution that can make Drupal work for an infinite number of environments in a more flexible way. ##Just the Beginning Anything that differs by environment can and arguably should be set as an environmental variable. If there's anything else you want to be different in your development environment than your production or staging environment (such as API credentials), consider creating a new environmental variable. Drop me a line in your comments to tell me how using environment variables is working for you and what you're using environment variables on your site! Site Review - BountyHunterWine.com 2013-10-20T00:00:00+00:00 BountyHunterWine.com offers a wide selection of wine for purchase online, but their user experience leaves something to be desired. Rather than bailing on their site, I decided to articulate the problems that I encountered and try to suggest constructive alternatives to the current implementations. This post is simply the contents of that letter. Dear Bounty Hunter Wine, I like the look of your elegant website and the prices are also good on your products. My purpose for using your website today is to purchase some wine for three business partners as a thank you gift. I want each business partner to receive one bottle of wine each with a customized "thank you" message. Each partner lives at a separate address. I was able to easily find the wine I wanted at a very reasonable price without any problems. I really like the "Price Range" filter. However, I've run into several problems trying to place orders on your site that I wanted to make you aware of. 1) SSL Certificates</h3> Your SSL certificates are only valid for www.bountyhunterwine.com which is fine. However, I was not redirected from bountyhunterwine.com to www.bountyhunterwine.com when I went to your site. This caused the "invalid certificate" logo from thawte to appear as well as warning messages that made your site look insecure. I recommend you pay your developers to setup such a redirection to create a trustworthy environment for your customers. 2) Adding Products to the Shopping Cart</h3> When I got to this screen I ran into trouble. I was only allowed to add 2 recipients at a time. Adding a third recipient caused the "Remaining Quantity" to read "-1" and I was unable to click "Continue Shopping" or "Viewcart" without triggering an error message. However, by starting over, I was able to add one wine bottle for 2 of my business partners, then I clicked "Continue Shopping." I returned to the product, and I was able to add a bottle to my shopping cart for my third business partner. It seems to me that there are more than 2 bottles of "2009 Stag's Leap Wine Cellar's Merlot" in your inventory, and that I ended doing a lot of extra work to place this order for all three of my partners. There are several ways of improving this part of the shopping experience. I think the best experience for me would have been to indicate the total stock of wine with "Remaining Quantity" and then to allow me to add recipients and quantities of wine ONLY if the actual remaining quantity was >= 0. Using a Javascript alert box if the remaining quantity actually reaches 0 would tell users that there's no more wine to be had. However, I don't understand the benefit of preventing users from ordering as much wine as we want to while additional stocks of wine are on hand. 3) Account Creation Form</h3> a. Privacy</h4> Once I add managed to add all the wine to the shopping cart, I started the process of creating an account. I would prefer not provide my phone number during this process. I seriously considered finding another wine store that didn't require a phone number. b. Address Fields</h4> I was very confused by the bottom half of this form. In particular, I would expect the address fields to match the order I write them as an address on an envelope, namely: Street 1 Street 2 City State Zip Country </code></pre> It's amazing how difficult I found it to fill in these fields when they were in a surprising order. c. Zip Code & City Fields</h4> Since I was confused by the order of the address fields, I entered the city first then the zip code. Doing so caused some JavaScript on the page to erase the value I provided for city. Also, the city field transformed from a text field into a dropdown list. However, I didn't notice this happen. d. Premature Account Creation</h4> After I pressed submit, the page informed me that I failed to fill in some fields on the form (namely the city field). I had to carefully examine each field to determine where the form needed to be filled in. Once I fixed the city field, I tried to submit the form again. This time I received the error message that a user on the site already had that email address. What happened was that the server side code had started processing the information from my first form submission without fully validating it. This is a serious security risk and a poor user experience. The server side code should instead be checking all form fields are: filled in (if they are required)</li> validated so that make sense (ex: zip codes should be 5 digit numbers)</li> sanitized to prevent attacks from hackers and evil internet robots (better known as "bots")</li> </ol> Only once everything is fully validated should the data provided by the user be added to the database or used by the shopping cart. I imagine other customers have this same problem that I experienced, and I believe that this is the most crucial and (and complicated) problem to address on your site of those that I discovered. 4) Adding Shipping Addresses</h3> My reason for shopping at bountyhunterwine.com is to purchase wine for my three business partners. I work very closely with only one of them, and unfortunately I don't have all of their phone numbers. I'm not sure if your shipping company requires a phone number for each destination address or not, but making the phone number optional would make it much easier for me to use your website. Also, I'd prefer not o divulge the personal information of my business associates without their explicit consent, and I was hoping to surprise them. CONCLUSION</h2> I hope that you find this feedback constructive. Let me know if you're able to make progress on these user interface and backend suggestions I've provided. Best of luck with your online business venture. Cheers Bryce Fisher-Fleig Ditching Views For SQL, Part 3 2013-10-13T00:00:00+00:00 In part one of this series, I discussed the reasons not to use Views</a> including the difficulty in debugging Views and the fact that Views is only in Drupal (whereas SQL is everywhere online). In part two, I walk you through some simple SQL queries and tools</a> that you'd use to find the data for a sample project. In this post, I'll connect the results of that SQL query to the Drupal theme layer so that we can actually use that dynamic content in our Drupal site without relying on Views. For today's lesson, you should at least be comfortable with some basic Drupal theming already. The End Goal</h2> As a quick reminder from last week, this blog post is a record of how I'm planning to replace the view on my homepage with SQL and Drupal theme hooks. So, the end result we want to have is a list of the most recently created blog titles, the body field, and the node id (so we can make links out of the title). We'll be taking the SQL we wrote last time and doing the preprocessing and theming today. The final markup should look something like this: <ol class="blog-articles"> <li class="article nid-1"> <a href="http://bryceadamfisher.com/blog/first-post"> <h3 class="article-title">First Post</h3> </a> <p class="article-summary">Shortened summary of the post</p> </li>  </ol> </code></pre>Hook_theme() 101</h2> After playing around with the Drupal theme layer for a while, I've really fallen in love with the Drupal theme hook. This hook is really powerfully, and I'm only going to show you my favorite way to use it, but you should know that this hook gives you all kinds of flexibility in case you would like to use it a slightly different way. After reading this tutorial, check out the API docs for hook_theme()</a> if I pique your curiosity. Some Background on Drupal Hooks</h3> Let's assume you're using a custom theme called myblog</code>. In Drupal speak, myblog</code> is the "machine name" of your theme. In practice, myblog</code> acts as a PHP namespace of sorts. We use this naming convention because PHP itself causes all functions to live in the same global namespace. So, Drupal has adopted a naming convention of starting all functions declared inside a theme or module with the name of the theme (myblog) to avoid namespace collisions. In addition, all the files associated with that theme will live in a folder called myblog</code>. Some function names are special in Drupal. Functions that use certain names are invoked by Drupal in response to specific conditions. These specially named functions are called "hooks." In the Drupal API docs, we talk about these functions as hook_theme</code> (for instance), or hook_install</code>, hook_enable</code>, hook_update</code>, etc., etc. Whenever you see something referenced as hook_whatever, substitute the theme (or module) namespace for the word hook. Thus, hook_theme</code> becomes myblog_theme</code>. A quick note on Drupal 8: Drupal 8 is a rewrite of Drupal based on the awesome Symfony project. Drupal 8 uses object oriented techniques to replace the hook system found in all earlier versions of Drupal. At time of writing this, Drupal 8 is scheduled to come out sometime late 2014. My hope is that using object oriented techniques will make debugging Drupal easier by providing easier to follow "breadcrumbs" about what order code is executed in. Getting Our Hands Dirty with Hook_theme()</h3> Inside your theme folder (/sites/all/themes/myblog), create or edit the file template.php. Inside this file, create a function myblog_theme()</code> like this: <?php function myblog_theme($existing, $type) { return array( 'front_blog_teasers' => array( 'template' => 'front-blog-teaser' ), ); } </code></pre> Here's the important points to note: This function must return a specially formatted array.</li> The array keys ('front_blog_teaser') can be anything we want, as long it hasn't already been used.</li> The value for template</code> defines which template file will be parsed, but the file extension is automatically set to ".tpl.php"</li> </ul> This function is only invoked during Drupal cron runs, and the results are cached. So, you must run cron each time you make changes to this hook. What Did We Just Do?</h3> We've done three important things with this hook: Registered new custom theming</li> Connected that custom theming to a specific template in a debug-friendly way</li> Made new preprocess functions available</li> </ol> We'll take advantage of each of these pieces to do everything that Views was doing for us earlier. 1. New Custom Theming</h2> Anywhere we want to display our dynamic content, we can invoke: <?php print theme('front_blog_teaser'); ?> </code></pre> If you're accustomed to working with Views, the equivalent code would be something like: <?php print views_embed_view('name_of_view'); ?> </code></pre> In this project for my blog, I'd place this code just under the $content</code> region in page--front.tpl.php. 2. Connected that custom theming to a specific template in a debug-friendly way</h2> In my opinion, one of the worst parts of Views is the God-awful template names that it forces you to use. I'm sorry, but views-view-unformatted--blog-article-block-3.tpl.php is a mystery to me. Especially as a site grows in size and complexity, you end with dozens of views templates. The file name for the template does not provide any insight into where you might expect that template to be used in the site. Views 2 for Drupal 6 tried to help by helping you track the templates being used inside in the Views Admin UI. However if the view templates are nested at all, even the Views Admin can't explain it to you. Kiss your afternoon goodbye trying to untangle the web of views templates. Using hook_theme</code>, we can do a text search for the name of the template across all files in the project. Such a search would show us where this template is registered, and we'd know to look for somewhere print theme('front_blog_teaser');</code> was written to see how this code ended up in the browse. There's no equivalent searchable, debuggable way to uncover what's going on in Views. What to put inside the template</h3> Let's actually write the template file code now. Inside the directory /sites/all/themes/myblog/, create the file front-blog-teaser.tpl.php</code>. Remember, we'll want to display any number of different post teasers on the homepage. Let's take the markup we want to end up with (above), and substitute imaginary variable names into that html. Later, we'll make sure these variables get injected just how we want them, but for now let's focus on the end result. Here's what I would do: <ol class="blog-articles"> <?php foreach($articles as $article) : ?> <li class="article nid-<?php print $article->nid; ?>"> <a href="<?php print url("node/{$article->nid}"); ?>"> <h3 class="article-title"><?php print $article->title; ?></h3> </a> <p class="article-summary"><?php print $article->summary; ?></p> </li> <?php endforeach; ?> </ol> </code></pre> Quick observations: An array of objects named articles</code> must be injected into this template</li> Each object inside articles</code> should have the properties, title, nid, and summary.</li> </ul> A Word On Debugging Templates</h3> I've chosen each and every variable name because it makes sense to me. However, you could change any of these variables. When debugging the equivalent code in a view template, you would have to use the variables that views forced you to use. Normally that might actually be a good thing, because then no matte who is maintaining this code, Views will always use the same variable names in the same situation. I often want to see what variables are available using var_dump()</code> to print the values and data types of an expression. With Views templates, the variables injected into the template have so much recursion that var_dump</code> will cause PHP to run out of memory and crash. This effectively makes me blind and unable to see what variables are available in the template. Simple changes can take days and have unintended consequences because I don't have good insight into the data that's available. By using preprocess functions, the data is dead simple. You can inject primitive data types and know exactly what to expect. If you want to change something, there's a trail in hook_theme</code> connecting the template, the custom theming name (ex: 'front_blog_teasers') and the template altogether. The time I spend programming, the more I come to rely on these sorts of breadcrumbs to solve the daily grind of problems that fall into my lap. 3. Made preprocess functions available for this custom theming</h2> This is the only piece of black magic you need to use. I saw black magic because there's no strong trail of breadcrumbs connecting this function to hook_theme</code>. You just have to know about this trick. (Although, if you read carefully through the API docs for hook_theme, you'll see how to explicitly spell out a preprocess function.) By default, the name of the preprocess function is themename_process_HOOK</code>. So, for our example, the theme name is myblog</code> and the HOOK is front_blog_teasers</code>. So, the preprocess function must be named myblog_preprocess_front_blog_teasers</code>. What's a Preprocess Function?</h3> In Drupal, registering a new custom theming in hook_theme automatically makes a "preprocess" function available. This is where you get to do any special logic to gather and prepare any variables needed in the template. You also get to name the variables and inject them into the template using references. So for our situation, we'll want to put the drupal SQL query we wrote in part 2 inside the preprocess function and then pass those variables back to the template. Here's how we do that: <?php function myblog_preprocess_front_blog_teasers(&$vars) { // Query all the raw blog teaser data $result = db_query(" SELECT n.nid, n.title, b.body_value FROM {node} n JOIN {field_data_body} b ON n.nid = b.entity_id WHERE n.type = \"article\" AND n.status = 1 ORDER BY n.created DESC "); // Format the raw data as PHP variables $articles = array(); while ($row = db_fetch_object($result)) { $row->summary = mb_substr($row->body_value, 0, 200); $articles[] = $row; } // Inject these variables into the template $vars['articles'] = $articles; } ?> </code></pre> There's only two changes here. You notice that we queried the database for the whole body value using SQL, but we're displaying the article summary in the template. I added this line of code to create the summary: $row->summary = mb_substr($row->body_value, 0, 200);</code>. The reason I did this was to demonstrate how you might prepare some special information for the template. I should point out however that there is a whole field in the database for the summary and you should probably you use that field instead since blog authors might use a different summary than the first 200 characters of their article. The other thing worth noting is how we injected the data into the templates. We simply use create an array key in the $vars variable. Since we used &$vars</code> as the parameter to this method, any changes to the $vars</code> variable will automatically be available as if we returned those changes. I don't love that feature of Drupal, but PHP only allows you to return one value for a function, so I understand the reasoning for this confusing convention. Note that every key in $vars array will a variable in the template. Save the files, clear the freaking Drupal cache, and you should have your html appear just as you wanted it to. I'll leave it to your imagination to setup the proper CSS styling for this project. Taking It Farther</h2> We could have done a lot more. We could used SQL to capture the number of comments associated with an article, the date, the author's name, and the image field for each post. Using the tools and techniques in parts 2 and 3, you easily accomplish this on your own. You could also leverage the Drupal caching system to store the final template markup in your preprocess function. The best way to take this farther is to experiment with the other possibilities using hook_theme. We've just barely scratched the surface! Conclusion</h2> Although it takes more time to setup, using SQL and hook_theme()</code> allowed us to write easily debuggable and understandable code with much less CPU overhead than running Views. You'll thank yourself at 1am when you need to fix something fast on the homepage that you didn't use Views. Ditching Views For SQL, Part 2 2013-10-09T00:00:00+00:00 I'm going to dive into some actual code in this post to demostrate Drupal without Views. I'm only going to cover how to write the SQL — I'll cover the theming functions in a third part. I'm starting from the assumption that you're already comfortable with SQL (particularly select statement and joins). If you don't understand what I just wrote, look up tutorials on those topics before trying to read here. Database Abstraction Layer in Drupal</h2> The typical pattern for getting data out of the database using SQL is: Examine the database structure</li> Write a working SQL query</li> "Drupalize" the sql syntax</li> Pass the query through db_query()</code></li> Retrieve query results using while</code> and db_fetch_object()</code></li> </ol> For this article, we'll write a query that grabs the node ids, titles, and bodies for the last three blog posts. Let's imagine I'm rewriting my homepage view. So we'll want the blog posts in date order, starting with the most recent at the top of the list. At the end of this post, our code should return an array of three each objects with the properties nid</code>, title</code>, and body</code> ordered by created date. 1. Examine the Database Structure</h3> There's a lot of tables in Drupal. Usually finding the fields we want involving poking around in the database. Never poke around the live database since you could ruin your site. Instead, download a copy of the database to your own computer and look at the database in your favorite MySQL client. If don't have one already, I highly recommend HeidiSQL for windows. You could PhpMyAdmin, but I find it slow and complicated compared to a desktop application like HeidiSQL. After poking around in my database, I found that the node</code> table contained almost everything I needed: the content_type, title, nid (node id), and created. The only thing missing was the body field. 2. Write a Working SQL Query</h3> Let's ignore the body field for a moment just get most of our query written: SELECT nid, title FROM node WHERE type = "article" AND status = 1 ORDER BY created DESC; </code></pre> If you run that query in the "query" tab on HeidiSQL, you should see three results in the correct order with the nid and title fields. The only thing to explain here is that status = 1</code> clause filters out unpublished nodes. So far so good! Okay, let's add in the body field now. Body Field Drupal 6</h4> If you're on Drupal 6, the body field is always stored in the node_revision</code> table. Just join the node</code> and node_revision</code> tables on the vid field. Here's the SQL for Drupal 6: SELECT nid, title, body FROM node JOIN node_revision ON node.vid = node_revision.vid WHERE type = "article" AND status = 1 ORDER BY created DESC; </code></pre>Body Field Drupal 7</h4> Drupal 7 is a little different. The body field is in the field_data_body</code> table, and the field is called body_value. Here's the SQL Drupal 7: SELECT nid, title, body_value FROM node JOIN field_data_body ON node.nid = field_data_body.entity_id WHERE type = "article" AND status = 1 ORDER BY created DESC; </code></pre>3. Drupalize the SQL Syntax</h3> Drupal's database abstraction layer allows you to automatically prefix your table names in the queries. (It also allows parameterized queries which is vital to security, but it doesn't apply to our situation). Doing this step makes your SQL more portable and is really easy. All we need to do is wrap curly braces around the table names. By convention, queries also use aliases for table names. Here's the SQL for Drupal 6: SELECT n.nid, n.title, r.body FROM {node} n JOIN {node_revision} r ON n.vid = r.vid WHERE n.type = "article" AND n.status = 1 ORDER BY n.created DESC; </code></pre> Here's the SQL for Drupal 7: SELECT n.nid, n.title, b.body_value FROM {node} n JOIN {field_data_body} b ON n.nid = b.entity_id WHERE n.type = "article" AND n.status = 1 ORDER BY n.created DESC; </code></pre>4. Pass the query through db_query()</code></h3> For brevity's sake I'll show you the Drupal 7 version only: <?php $query = "SELECT n.nid, n.title, b.body_value FROM {node} n JOIN {field_data_body} b ON n.nid = b.entity_id WHERE n.type = \"article\" AND n.status = 1 ORDER BY n.created DESC"; $result = db_query($query); </code></pre> It's important to keep the return value of db_query()</code> so that we can actually pull out the individual rows from the database. 5. Retrieve Query Results Using while</code> and db_fetch_object()</code></h3> Here's how this bit works. db_fetch_object</code> is given the $result of a previously run query. If there is another row among the results, db_fetch_object</code> will return an object with each field of the SQL result as a property on the object. In our example, the object will have three properties: title, nid, and body. However, if there's no more results from the query, then db_fetch_object()</code> returns false. In PHP, if you assign a "falsy" value to a variable, that whole expression evaluates to false. This is really handy in while loops. Again, I'm only showing the Drupal 7 version for brevity (db_query()</code> and db_fetch_object()</code> are pretty much the same in those versions of drupal). This code follows the code above: <?php // Store the results here. $nodes = array(); // Continue pulling out result rows from the query results, // but stop if there are no more. while ($row = db_fetch_object($result)) { $nodes[] = $row; } //Output the result to the browser var_dump($nodes); </code></pre> Again, the only thing to really explain here is that when db_fetch_object reaches that last row in the result set, it will return false. Setting $row to false makes everything inside the while conditional evaluate to false. That kills the while loop. Simple and elegant! Conclusion</h2> In this article, we've walked through some tools to help discover the layouts of tables in Drupal (don't be afraid to explore locally!), written a working SQL query in HeidiSQL, then ported this query to PHP/Drupal code. Ready to implement these features in the theme layer? Keep reading part 3 in the Ditch Views for SQL saga</a>. Ditching Views For SQL 2013-10-07T00:00:00+00:00 I've been using Views in Drupal since I first started using Drupal. It's easy to create dynamic content that stays up to date throughout the site, and on, and on, and on. It's the most comprehensive SQL query builder that I've used, and it's a truly magical piece of software. The Drupal community owns a lot to Merlin of Chaos and the legion of Views contributors. What's Wrong with Views?</h2> Debugging. Like many parts of the current Drupal 7 (and Drupal 6) ecosystem, Views works like a charm until something goes wrong. At my 9 to 5 job today, I ran into a situation where I needed to change my underlying content type and now my template started displaying random fields. No problem? Views creates a whole host of wonderful preprocessed and postprocessed variables available in the template. So, like any good themer, I var_dump()</code>'d the $node object in search of the data I wanted. A quick search through the output showed me that the string was available somewhere inside the $node object. Hooray! Problem solved. Sadly, this was on line 8907, and I couldn't see how to drill down to this variable from the $node object. I decided to write a recursive function to search through all properties of the $node object, and when I left work two hours later, I still hadn't gotten this recursive function quite right. Certainly, I could stick with Views and find a way to output the raw field value. The Heart of the Problem</h2> However, let's take a step back and think through what I'm doing here. Views is supposed to prevent me from having to deal with the complexity of writing SQL queries and formatting the outputting. Click, click, click, and you're done. The problem is that while I actually understand SQL fairly well, I don't have any idea what's going in Views when things break. And, due to the unusual complexity of the Views module, it's many plugins and API and recursive objects, it would take me days to troubleshoot even a relatively simple problem like this. Don't get me wrong, I like learning new things, but this is a not a good use of my employer's time. The heart of the problem is that Views, while catering to those who are unable to write SQL, is creating overwhelming complexity for everyone. Part of the reason for this is that Views tries to be comprehensive. It features a whole new theme layer, new url routing logic, extra access control, a suite of plugins to format raw field values, live previews, import/export from code, and a whole bunch of other goodies. In a word, Views is trying to do too much. The Right Way to Approach SQL</h2> With my degree in philosophy, I tend to be shy about using the "R" word, but I think we can safely say the "normal" and sane way to ease the burden of writing SQL is to create Object Relational Mapper (ORM). An ORM is an internal API for accessing the database. In CakePHP, the ORM is the first thing you learn</a>, about CakePHP after doing the installation. Similarly, it's impossible to do much in Code Igniter or Ruby on Rails without leveraging an ORM. Drupal 7 has taken some steps in this direction with the EntityFieldQuery</a> class, but it's probably the last thing you learn about in Drupal. Also, it tends to be a little bit confusing. For my day job, I'm currently stuck with Drupal 6, so I don't even have the EntityFieldQuery class available. Rather than languish in despair, I actually find the SQL quite easy to write. In all honesty, rewriting the code that tripped me up today in SQL is probably faster than figuring what went wrong in my views template. If you don't know SQL yet, consider taking the time to learn it. Unlike Views, you can use your experience with SQL outside the Drupal community and even with other programming languages like Python/Django, or Ruby/Rails. I think you'll find that this is a good investment in your career as a programmer. Up Next</h2> Ready to make the shift? Check out Ditching Views part 2</a> to learn about how to implement my advice on your own drupal instance. The Missing Guide to Subnet Masks 2013-10-05T00:00:00+00:00 Recently, I tried to understand what was going on in terms of IP ranges in my firewall configuration, and I had an impossible time trying to find a succinct and clear explanation. This post aims to make it a little easier to understand how to read and specify ranges of IP addresses using subnet masks. Sexy Subnet Masks in Binary</h2> Subnet masks are often used by system administrators to specify ranges of IP addresses in a compact format. In this context, the word mask means a binary number used to transform an IP Address using boolean algebra (AND, OR, XOR operations). While subnet masks might not be sexy, seeing the binary form of a subnet mask is the only way to visualize what's going on. Let's start with an example subnet mask: 11111111 . 11111111 . 11111111 . 11110000 </blockquote> Notice that the subnet mask consists of 4 octets of bits. This mask is used to transform an IP address. So let's take an imaginary IP address, 50.122.34.21, to play with. We'll need to convert this address into 4 octets of bits so that we can easily visualize what's going on: 00110010 . 01111010 . 00100010 . 00010101 (IP Address) </blockquote> Straight Up Boolean, Baby</h2> All subnet masks use boolean AND to combine the IP Address and the mask. Let's review how boolean AND works: 1 AND 1 = 1. 0 AND 1 = 0. 1 AND 0 = 0. 0 AND 0 = 0. </blockquote> Okay, now putting that altogether, let's AND the IP address and the subnet mask together: First Octect</th> Second Octect</th> Third Octet</th> Fourth Octect</th> Name</th> </tr> </thead> 00110010</td> 01111010</td> 00100010</td> 00010101</td> IP Address</td> </tr> 11111111</td> 11111111</td> 11111111</td> 11110000</td> Subnet Masks</td> </tr> 00110010</td> 01111010</td> 00100010</td> 00010000</td> Result</td> </tr> </tbody> </table> </div> Notice how all the ones on the left side of the subnet mask allow the starting IP address to trickle down into the result. Only the rightmost bits (where the mask has 0's) are altered. At the end of the process, we get 50.122.34.16 converting the binary result back into decimal. A Subnet Mask By Any Other Name</h2> So, why would we care about getting out 50.122.34.16? We care because we now have a way of saying "do something to all the IP addresses between 50.122.34.21 and 50.122.34.16". Let's take this a little farther and make subnet masks more useful. Look at the binary form of the subnet mask once more: 11111111 . 11111111 . 11111111 . 11110000 </blockquote> Notice, that all the 1's are on the left and all the 0's are on the right. This is no accident. Subnet masks are not allowed to put ones and zeros anywhere you like. Some number of 1's must go on the left side, and the rest of the bits will be zero. Since this is the case, you could abbreviate the subnet mask by just counting the number of 1's. For instance the above subnet masks could just as easily written as: /28 </blockquote> Likewise, these two notations are also equivalent: /8 11111111 . 00000000 . 00000000 . 00000000 </blockquote> So, combining this short form of the subnet mask and the original IP address could also be written like: 50.122.34.21/28 </blockquote> While this form is harder for mere mortals like myself to read, I suspect that this format is much much faster for networking devices to compute and store. It also has the advantage of being much shorter than writing out the full version of two IP addresses. In the literature on this topic, you often find the decimal notation for subnet masks as well. For instance, /28 would be written out as 255 . 255 . 255 . 240. Personally, I find this notation unhelpful, so I've avoided it in this article, but you should expect to see it written this way sometimes. The Gotcha</h2> You knew it was coming, didn't you? Notice that our transformed IP address from the example above (50.122.34.16) ends in 16. This too is no accident. Since our subnet mask uses binary numbers, the transformed result will nearly always end in a power of two. This makes starting or ending an IP range on an arbitrary number between powers of two hard, because you'll need to break down the range into a series of more binary friendly sub-ranges. I'll leave you play around with the math yourself. There are some great subnet mask calculators online</a> that can help play with the examples above or your own firewall configuration to better understand the concepts in this article. Conclusion</h2> Firewalls often use subnet masks in the compact form IP/number of 1's to specify a range of IP addresses. Understanding the boolean AND operation is the key to understanding how these subnet masks work. 7 Steps to Improving Page Load Time on Shared Hosting 2013-06-08T00:00:00+00:00 I recently optimized page load time for the Spelling Word's Well homepage</a> as part of a redesign. Using the YSlow Add-on</a> helped, but I still had a learning curve to face, and I also uncovered some other tricks. Here's a summary of the most important steps I took: 1. Make a DIY, Cookie-free Content Delivery Network</h2> The key ingredient here is to use a subdomain that delivers all your static content: images, css, static json files, xml dumps, javascript, etc. So, you'll need to assemble all those assets if they are scattered around. I often have cPanel available on my shared hosting sites. Log into cPanel and click Subdomains in the Domains box. Choose a subdomain name that makes sense to you. I like to store these files outside of the public directory of the main domain (hence /subdomains/static instead of /public_html/static in the screenshot). Once this directory is created, be ruthless about putting everything you can on this subdomain, and update your html to use this subdomain. UPDATE: Finally, make your DIY CDN real using CloudFlare for free</a>. UPDATE April, 2014: See my walkthrough describing how to setup AWS CloudFront with a custom SSL cert and an AWS S3 bucket</a> (what I do on this site) for as little as $10 for the 1st year. 2. Supercharge Your Subdomain</h2> Create an htaccess file in the root directory of your subdomain (in my example, /subdomains/static/.htaccess). The following .htaccess file asks browsers to store images and other static assets in it's cache longer for better repeat traffic. It utilizes mod_deflate to compress all kinds of text files (js, xml, html, css, ajax, etc.) over the netwok. It also blocks access to directory listings and sensitive batch files for security. # Security Options -Indexes <IfModule mod_rewrite.c> RewriteEngine On RewriteRule \.bat$ - [R=404] </IfModule> # Leverage Browser Caching <IfModule mod_expires.c> # Enable Expirations ExpiresActive On # Default Expiration Time ExpiresDefault "access plus 1 month" # Expiration for Images ExpiresByType image/gif "access plus 1 month" ExpiresByType image/png "access plus 1 month" ExpiresByType image/jpg "access plus 1 month" ExpiresByType image/jpeg "access plus 1 month" # Expiration for CSS ExpiresByType text/css "access plus 1 month" # Expiration for JavaScript ExpiresByType application/javascript "access plus 1 month" </IfModule> # Gzip <ifmodule mod_deflate.c> AddOutputFilterByType DEFLATE text/text text/html text/plain text/xml text/css application/x-javascript application/javascript text/javascript </ifmodule> #End Gzip </code></pre>3. Fight the Scourge of Social Sharing Widgets</h2> Seriously, this is the most important thing. Each facebook like and google sharing widget REALLY slows down the page. To use these widgets, the browser has to look up the IP address of the domain, contact the real server, ask for the javascript, run the javascript, lookup another domain's IP address for the images, ask for the images, download the images, and then display them. It's all horribly inefficient. It's important to weight the lost conversions on your site against the possible the sharing benefits of showing the widget. Not everyone will share your page, but everyone will have to suffer through another agonizing second of page load time for every widget on every page. So, how do you fight back? Reduce unnecessary social widgets On Spelling Words Well, I removed a facebook badge that was hiding down the page, and saved almost a second in page load time.</li> Load Widgets using AJAX Many widgets do this already, but check the documentation for more info.</li> Look for simpler alternatives You can download the images, javascript, and html and server from your own CDN and avoid a lot of the overhead of googleplus and facebook widgets without sacrificing the ability to share. Thank you Twitter!!! (Also, your designers can go to town on the sharing widgets).</li> </ul> UPDATE: In a future post, I'll walk you through using twitter web intents in a more detailed way, and I'd like to discuss creating customized performance focused google plus sharing widgets that function similarly to the twitter web intents. Follow me on twitter</a> to find out when I write this follow up article! 4. Optimize Images</h2> There's two parts to optimizing your images. The (more) obvious part is that you should use the lowest quality settings for JPGs that you can bear to look at. This makes the filesize smaller without shrinking the image's display size on screen. PNG files can be compressed using the free GIMP</a> photo editing wonderland software. Don't use gifs. Just don't. UPDATE October, 2013: I now recommend the incredible (and currently free) service from kraken.io</a> to shrink images an exta 10-25% on average. Kraken does this without sacrificing any image quality. You can drag and drop dozens of files and download the optimized images as a single zip file once Kraken has worked it's magic. The somewhat less obvious part is to crop your images in your photo-editing software (or in Drupal or Wordpress) so that the browser won't have to resize the image on fly. Setting a size in the <img /></code> tag that is different the size of image stored in the server will force the browser to do extra work to resize the image. While is this really cool, it is slower. Of course, if you're using a responsive design, you'll have to weigh this carefully, and probably disregard my advice. Responsive images have a whole host of other puzzles to untangle that requires several posts to tease out. 5. Make the Page Beautiful Without JavaScript</h2> First of all, Google is your biggest fan, and Google doesn't use javascript when it's examining your site. Also, a small percentage of your visitors will not be using JavaScript either, so you want to make sure those folks have a nice experience. The main thing to do is to place something simpler but useful where the JavaScript content was. Make sure all links work, and everything makes sense with JavaScript disabled on your page. This could also be the subject of a whole post. 6. Make JavaScript Run Last</h2> JavaScript is the single slowest part of any page, and while it's loading in the browser it "blocks" the browser from doing anything else. Therefore, if you've made a website that already works without JavaScript, just make JavaScript run last. This way, your users can start interacting with the page right away, and the page will get better once it's fully loaded. Here's what to do: Move all JavaScript code to the bottom of the page, right before the closing </body></code> tag.</li> Move all that code into a separate JS file, and point a <script></code> tag at the new JS file</li> Surround all your JavaScript with an onload callback function (such as jQuery's $(document).ready()</code> or the built in javascript window.onload=function(){}</code>.</li> Combine all javascript files into one file if possible (jQuery and plugins, facebook, googleplus, your own code, etc). Be careful about the order of things! For example, make sure that jQuery is added before any jQuery plugins.</li> Experiment with different orders to see what feels the fastest</li> </ul> If you did steps 5 and 6, you'll start to notice your page feels much much more responsive already. Well done! This is the hardest part. 7. Minify CSS and JS</h2> I've used YUI CSS Compressor</a> and been happy with my compressed css. Once you're happy, upload it to your own CDN from step 1. Alternatively, if you've started using LESS, SASS, or Stylus, these all can minify the css they produce for you easily. Consult the appropriate documentation for details. Personally, I like using Google Closure Compiler to minify my JS. I've setup a batch script that includes all my JS files in a specific order and compresses it into a slightly smaller output file. That should be the subject of a whole separate post later, but suffice it to say that you can minify using closure compiler online</a> too. Just the Beginning</h2> There's a lot that could be said about image sprites using Compass</a>, and I've only scratched the surface of JavaScript optimizations that can be made with social media. YSlow also offers plenty of other advice which is likely to help you a lot. Pitfalls while Optimizing JavaScript 2013-04-30T00:00:00+00:00 I've tried to incorporate some best practices from JavaScript Patterns by Stoyan Stefanov of Yahoo! Press. Here's some pitfalls I've run into and some simple fixes. The examples are straight JavaScript, no libraries used at all. However, the concepts and techniques could easily be applied to jQuery, Dojo, or other libraries, especially the for-loop section. Variable Hoisting</h2> JavaScript essentially moves any variable declarations into the top of the current function (or global namespace), regardless of it's placement in the source code. There are some edge cases where this makes variables be undefined or have confusing values. There's some possible value in having all variables defined in the same place in your code (from a readbility perspective). If you're working with hundreds of lines of code, or you have a very complex app you're building, this is definitely a best practice. For short snippets of code, this is probably more optional. However, I'll be following this practice in the sections to follow. There's no real pitfall here, just an FYI! Optimizing For-Loops</h2> Cache the length</code> property</h3> An easy way to speed up JavaScript code is to optimize for loops. The length</code> property of arrays is actually more like a function call in that the browser will have to rexamine the array each time your script accesses the length</code>. The way to optimize this situation is to read the length of the array once, store it in a local variable, and then compare the iterator to the local variable. Here's an example: var myArray = [1,2,3,4,5,6,7,8,9], max = myArray.length; for(var i=0; i<max; i++){ console.log(myArray[i]) } </code></pre> Simple, right? Using max</code> prevents the browser from reexamining myArray</code>'s contents on every iteration of the loop, thus avoiding unnecessary work. This effect becomes more pronounced on larger arrays. Count Down, Not Up</h3> While the first example above is all well and good, we could go a step farther. All else being equal, it's faster to compare a number to 0 than to another number. So, we can (in theory) speed up this for-loop just a little bit by starting i</code> at the length, and comparing to 0 and decrementing i</code> instead. The added bonus is that we get to remove an extra variable. Here's an example: //antipattern var myArray = [1,2,3,4,5,6,7,8,9], i = myArray.length; for(; i>0; i--){ console.log(myArray[i]) } </code></pre> Notice that we're still caching the length</code> inside a variable. The odd thing about this syntax is that we set i</code> outside the for-loop and thus the first statement is just a semicolon. While strange to see, you'd have to admit that the code inside in the for loop couldn't get much shorter. Zero-Relative Issues with Arrays</h3> If you try to run the second example above, it won't work. Why not? The length property is 1 relative, but the array indices are zero relative. So, for instance myArray[1]</code> would be 2</code>. The very first iteration of the loop, the script tries access a non-existent element in the array since it uses the length</code>. The solution is simple. Set i</code> to length - 1, and then inside the for loop update the conditional to include 0. Here's final code: var myArray = [1,2,3,4,5,6,7,8,9], i = myArray.length - 1; for(; i>=0; i--){ console.log(myArray[i]) } </code></pre>Optimizing DOM Lookups</h2> The Ethics of Cloning</h3> You can speed up your script to updating larger swathes of the DOM at a time whenever the nodes you're changing share a common parentNode</code>. It may not be appropriate for all situations, but I feel like I'm always iterating over a list tag and updating each of the list items. To implement this "pattern," grab the element you want, clone it, modify it, and swap out the original with the clone. Let's look at an example: //antipattern var old_node = document.getElementById('product-list'), new_node = old_node.cloneNode(), lis = new_node.getElementsByTagName('li'), i = lis.length - 1; for(; i>=0; i--) { lis[i].style.color = "red"; } old_node.parentNode.replaceChild(new_node, old_node); </code></pre>Copy the DOM Subtree Separately</h3> Unfortunately the above snippet doesn't work as you'd expect. The problem is that the W3C specifies that cloneNode</code> will only copy the dom object and it's attributes. However, the innerHTML, any text inside or nodes underneath are NOT copied. Thus, you'd get an exact copy of that node but nothing else. The solution is to copy the innerHTML property as well. Here's my example (you're welcome to improve upon it): var old_node = document.getElementById('product-list'), new_node = old_node.cloneNode(), lis = [], i = 0; new_node.innerHTML = old_node.innerHTML; lis = new_node.getElementsByTagName('li'); i = lis.length - 1; for(; i>=0; i--) { lis[i].style.color = "red"; } old_node.parentNode.replaceChild(new_node, old_node); </code></pre> Feel free to share your thoughts! Comparison of Drupal Modules for Responsive Images 2013-04-14T00:00:00+00:00 Drupal has a whole host of modules that are designed to make it easier for Drupal themers and site builders to create responsive images. In other words, these Drupal modules aim to make mobile browsers download smaller images (and load more quickly) while still serving up full sized images to laptops. The feature chart below is based on descriptions from the drupal responsive image module pages, and my own investigations. All modules are Drupal 7, unless otherwise noted. Responsive Image Features Explanation</h2> Module Name - links to the module's project page on Drupal.org.</li> Fallback? - What happens when JavaScript is disabled on the browser?</li> Affects Image Types - Descibes how images must be stored and retrieved from Drupal for this module to work on them. Does this module work on inline image tags, only image fields, or other types of images?</li> 3rd Party Libraries - Does this module require you to download and install code outside of Drupal.org?</li> Ease of Installation - "Easy" means simply turn on the module and possibly configure a few settings.</li> Integrates with - Lists key modules that tie into this module.</li> Maintenance Concerns - Issues to think about after initial setup of the module.</li> Reported Installs - How many sites report using this module accoding Drupal.org project page.</li> </ul> Comparison of Features</h2> Module Name</th> Fallback?</th> Affects Image Types</th> 3rd Party Libraries</th> Ease of Installation</th> Integrates with</th> Maintenance Concerns</th> Reported Installs</th> </tr> </thead> Adaptive Image</a></td> ??</td> Fields only</td> None</td> Easy</td> Image Styles</td> None</td> 4000ish</td> </tr> <tr> <td><a href="http://drupal.org/project/cs_adaptive_image">Client Side Adaptive Image</a></td> <td>noscript tag sets a fallback image</td> <td>Fields only</td> <td>None</td> <td>Easy</td> <td>Image Field</td> <td>None</td> <td>1500</td> </tr> <tr> <td><a href="http://drupal.org/project/ais">Adaptive Image Styles</a></td> <td>Original image</td> <td>Fields, and Inline Img tags</td> <td>None</td> <td>Hard</td> <td>Image Styles, Media, WYSIWYG</td> <td>Patching .htaccess module after drupal core updates!</td> <td>1700</td> </tr> <tr> <td><a href="http://drupal.org/project/resp_img">Responsive Images</a></td> <td>??</td> <td>Fields only</td> <td><a href="https://github.com/filamentgroup/Responsive-Images/tree/cookie-driven">Responsive Image library</a></td> <td>Moderate</td> <td>Image Styles</td> <td>NOT ACTIVELY MAINTAINED</td> <td>106</td> </tr> <tr> <td><a href="http://drupal.org/project/resp_img">Responsive Images and Styles</a></td> <td>CSS Based / JS independent</td> <td>Fields, Blocks, Entities textfields, Inline img tags, and more</td> <td>None</td> <td>Easy to Moderate</td> <td>Image Styles, Context, Media, Picture, field_slideshow, colorbox, expire</td> <td>Will be part of D8 Core, but breaks on Drupal 7.20+ without some TLC (see project page)</td> <td>1811</td> </tr> </code></pre></tbody> </table> </div> Concluding thoughts on Responsive Images and Drupal</h2> Given the set of features and integration with Drupal 8, I highly recommend going with the Responsive Images and Styles module listed above. It appears that it can do everything the other modules do and more, plus it's not much harder to configure (depending on if you use the Media module). However, if you want something simple with a larger set of users, Adapative Image would be my second choice. I would absolutely avoid Adaptive Image Styles given the hassle of potentially fixing the htaccess file every time you update Drupal core. I would also avoid the Responsive Images module since it is no longer maintained. Removing a Drupal Site 2013-04-11T00:00:00+00:00 I run a lot of development sites on my web server, and after the sites launch I remove the development sites. However, running a simple rm -rf drupal/path/</code> from the shell never works. There's always a few files that linger. $ chmod 777 sites/default sites/default/files sites/default/.php $ rm -rf sites/ </code></pre> During site-install, Drupal 7 protects 4 key files by setting permissions to something like 500. All we have to do is change the permissions and then delete them. Once you've logged in SSH, and changed into your Drupal root directory, follow above steps. Running PHP 5.2 and 5.3 on the Same Server 2013-01-11T00:00:00+00:00 In this tutorial, I'm going walk you through how to setup PHP 5.3 and PHP 5.2 (or other versions) on the same shared hosting account with HostGator. Obviously, having two separate servers would be the ideal scenario, but when budgets don't allow for that, it is possible to run different PHP versions on the same server and even account. Onward! UPDATE MAY, 2014 -- HostGator is now converting all PHP to version 5.4.</a> This is awesome because 5.4 is significantly faster than previous versions and it comes with a development server, traits, JavaScript array notation (ex: [1,"string",TRUE,[],...]</code>) and loads of other goodies and security updates. Great job, HostGator! ##Why??? I was running Drupal</a> as the CMS on my personal shared hosting account, and I've been playing with around with Fat Free Framework</a> for building a lightweight backend and RESTful API to a mobile app. Before dishing out the whopping $50/month to rollout my app on a dedicated server, I just wanted to do some basic user testing early on in the process. So, to save some money, I decided to use my personal shared hosting account (with HostGator). Unfortunately, Fat Free Framework requires PHP 5.3+ to work it's magic, but HostGator by default uses PHP 5.2. After puttering through their documentation, I discovered that PHP 5.3 comes pre-installed on all shared hosting. Hooray! However, none of the Zend extensions are compatible with HostGator's PHP 5.3 setup, such as PDO, sqlite, pdo_mysql, and others. Here's the rub -- Drupal depends on these modules being loaded. So how can we escape this from this most heart-rending dilemma ever seen??? ##Two Php Configurations Because HostGator uses suPHP by default, we can instruct suPHP to load different php.ini files inside an .htaccess file. So, once we know what directives need to be set for each platform, we can put a custom php.ini and custom .htaccess file in their directories. Since most .htaccess directives apply to all subdirectories, we only need to put these configuration files into the top directories that need PHP 5.3, and everywhere that PHP 5.2 or Zend is needed, we can just leave everything as-is. The end result will be that each project has exactly the environment it needs. To illustrate, here's how I've structured my home directory: /home/user/ php.ini # This is the default that comes with shared hosting. No need to change this! public_html/ # Drupal lives in here. Uses PHP 5.2 with Zend modules by default php53/ .htaccess # Tells Apache to use PHP 5.3 in this folder and all sub-folders php.ini # Tells suPHP not to load Zend modules </code></pre> ###Htaccess File for PHP 5.3 So, let's put together the htaccess file for PHP 5.3. First we need to instruct Apache to use PHP 5.3 in this directory and all other subdirectories: AddType application/x-httpd-php53 .php </code></pre> That was easy, right? Okay, now let's tell suPHP where to find our custom php.ini file: <IfModule mod_suphp.c> suPHP_ConfigPath /home/user/php53 </IfModule> </code></pre> NOTE: You'll need to change the path to point to the directory where you're custom php.ini will be. The example above uses the directory structure I've outlined earlier in the article. ###php.ini File for PHP 5.3 By default, HostGator puts a php.ini for you in your home directory. Copy that file into the directory where you want to use PHP 5.3; for this example that would be /home/user/php53</code>. Open this file in your favorite text editor, and comment out all the lines underneath [Zend]</code>. For me, this happens starting on line 1125, but it may be different for you. To comment out a line, simply put a semicolon ;</code> in front of it. Also, PHP 5.3 seems to require a timezone to be set in the php.ini file. Simply add this code to the very bottom (substituting the "America/Los_Angeles</code>" with your time zone</a>): [Date] date.timezone="America/Los_Angeles" </code></pre> When you are finished, the end of your custom php.ini file should look like this: [Zend] ;extension=magickwand.so ;extension=imagick.so ;extension=mailparse.so ;extension=pdo.so ;extension=pdo_sqlite.so ;extension=sqlite.so ;extension=pdo_mysql.so ;extension=uploadprogress.so ;extension=gnupg.so ;extension=mailparse.so ;extension=fileinfo.so ;extension=mongo.so ;extension=http.so ;extension=phar.so ;extension="ixed.5.2.lin" ;zend_extension="/usr/local/Zend/lib/Optimizer-3.3.9/php-5.2.x/ZendOptimizer.so" [Date] date.timezone="America/Los_Angeles" </code></pre>Making Sure It Works</h2> Once you've uploaded your custom .htaccess and php.ini to the proper folder, create a file called phpinfo.php, and add this to it: <?php phpinfo(); ?></code>. Upload this folder to a publicly viewable location inside /home/user/php53</code> and open it in your browser. Scanning through the output, you should have PHP 5.3 running, and you should find your custom php.ini mentioned as well. Once you've found this information, remove the phpinfo.php. You should now be ready to upload your PHP 5.3+ code into the folder. Further Reading:</h2> http://stackoverflow.com/questions/2271910/edit-htaccess-to-load-php-extension</a></li> http://support.hostgator.com/articles/specialized-help/technical/what-is-php-ini</a></li> http://support.hostgator.com/articles/hosting-guide/hardware-software/php-5-3</a></li> </ul> Debugging A PHP Error 2012-09-24T00:00:00+00:00 I classify any PHP error in two flavors: coding errors and design errors. Firstly, I'll discuss what a coding error is, why it might occur, how to find the error, and common fixes. Secondly, I'll compare and contrast with design errors, since they can be more subtle and complex. ##Coding Errors Coding errors involved a problem with the actual code written -- for example a misspelled word, a missing semicolon at the end of a line, etc. When configured to do so, PHP can provide error messages when it encounters a coding error, but often times error messages are turned off. However, in order to solve coding errors, you will need to configure PHP to provide you with these error messages. Why Are Error Messages Turned Off?</h3> For security. PHP error messages often display part of the PHP code used to operate your website, or provide detailed information about the inner workings of your code or your webserver. Online predators could use this information to infect your server, gain unauthorized access, do other unpleasant things. You can make your website considerable more secure from these kinds of risks by keeping PHP error messages hidden from view on any publicly available websites. Finding Error Messages</h3> There are two ways to keep these error messages private, while still being able see the error messages yourself: 1. Only Display Errors on a Private Server</h4> Typically, you'll have a "development" server on your personal computer or a company server. This server should not be reachable by the general public online. If you're working alone, you'll want to setup a development server on your computer using Apache, MySQL, and PHP. There are easy to install preconfigured webservers for Mac OSX (mamp) and Windows (xampp) available free online. Usually, php developers will do the early phases of programming on a development server before they are ready to show their website to the world. Then they move all the code to a public webserver once they are done with the initial programming. To display error messages for a single file, include this code at the beginning of your code, right after <?php <?php ini_set('display_errors','On'); // Your existing code below here </code></pre>2. Log Error Messages on a Public Website</h4> PHP can be configured to write error messages to a file named error_log. It does not have a file name extension -- the full file name is error_log. Typically, this file will be created in the same directory that error occurred in. Error logs are usually hidden from public view, but they can be downloaded through FTP so that you can still read the error message. This way, you can keep your server a little more secure and still be able to know where PHP encountered an error. To turn on error logging for your server, find your php.ini file (use phpinfo() to help you locate this file), and make sure there is a line in php.ini that has log_errors=On </code></pre> If you see a line: log_errors=Off </code></pre> Change it to "log_errors=On", and restart Apache. Understanding Error Messages</h3> There are several kinds of errors that you will run into if you work with PHP for any length of time. Syntax Errors</h4> The most common errors are syntax errors, and the most common among these are missing semicolons from the end of the previous line, or missing brackets, or missing parentheses. Here is a list of all tokens that might occur in an error message</a>. Fixing Common Syntax Errors</h5> PHP error messages always include a file name and a line number. Syntax usually occur on the previous line. Open the file name in error message, go to the line number, and look carefully at the previous line, making sure: There is a semicolon at the end of the line (if necessary)</li> All parentheses and brackets open and close properly</li> All single quotes and double quotes have a matching pair</li> Every if statement has a closing bracket</li> Every function has a closing bracket</li> Every class has a closing bracket</li> Every for, while, foreach, and switch statement has a closing bracket</li> Every part of an if statement has an opening and closing parenthesis</li> </ul> If that list didn't solve your problem, try commenting out the entire file, and uncomment line by line until you figure where the error is occurring. Preventing Syntax Errors</h5> Over time, you are unlikely to stop making this mistake. Therefore, most programmers like to use a text editor that has syntax highlighting so that you'll see visual cues about syntax problems as you type. This way, you can notice these problems right away, instead of spending minutes or hours looking for them on your own. This kind of help will save you hours and hours of time, and make your coding experience much more enjoyable. Some tools automatically create matching pairs of brackets, parentheses, and quotation marks for you as you type. Other programs beep if you type a syntax error. Personally, I've been VIm for years because it's free to download and use, it's crossplatform, it's on every linux machine ever, and it has a very healthy ecosystem of users and plugins. VIm has plugins like syntastic</a>. No tool is perfect, but most text editors will help you spot 99% of these kind of syntax errors. Design Errors</h2> Design errors occur when the code simply does not behave as expected, but no error message comes from PHP. Typically this occurs when the programmer (you!) has a wrong understanding of how some piece of software works. This can be a much harder to problem to solve. I use three different tactics to overcome a design error. 1. Reread the documentation carefully</h3> Often, I've not read the documentation slowly that explains the php code I'm using, or I've missed an important concept on how this software is supposed to work, or what feature it offers. All native php functions can be found by going to http://php.net/function. For instance, the var_dump</code> function's documentation can be found at http://php.net/var_dump</a>. Also, googling "how to do something using something else" can uncover unofficial documentation, or posting questions on Stack Exchange can uncover help from the community. Sometimes, documentation is out of date, incomplete, or just plain wrong. In these cases, it's time to try something else. 2. Try Another Approach</h3> Most of the time, you just want to accomplish some task and it doesn't make sense to spend too much time figuring why something isn't working, or how something new is supposed to work. If you've worked with PHP before, use an easier or simpler solution. Better yet use a technique or piece of software with which you're more familiar. Playing to your strengths is a great way to increase your own productivity. Sometimes you just have to get something very specific to work a certain way. In these cases try the next technique. 3. Verify each step of your code</h4> Use the var_dump()</code> or exit()</code> function to see what's happening at critical moments in your code, such as checking what arguments were passed to your function, or if a function is being called at all. This is best way to get familiar with a small section of code that is doing something strange. The trick is to figure out where you STOP understanding what's going on, and try to test different variables and values in the code to understand what's happening there. Don't assume anything! Keep finding and figuring out roadblocks one at a time until you've worked through the problem. There are several tools that can help you "step" through the code to check variables at point in the execution of a script. Perhaps the most popular tool is xdebug</a>. While this is a great way to understand code, it's very time consuming if you're working with a large piece of software like WordPress, Joomla, or Drupal. However, if you're a novice programmer or a PHP newbie, this is the best guide to learning PHP there is.

Running Gui Apps from Docker 101

Xeyes</h2> We'll use the xeyes application as a simple test that things are working. A docker image exists for this at fr3nd/xeyes</code>.</p>

Getting Fancy</h2> Now to put a cherry on top, we'd like to create a desktop icon we can use to launch the crazy eyes application without doing arcane docker magic. After much fumbling, I've discovered this magic:</p>

Helpful Source Material</h2> Nassos Michas post</a></li> This stackoverflow post</a> concerning permissions</li> This very specific stackoverflow post</a> concerning eliminating terminals from showing up</li> </ul>

How I Nvim-from-Scratch

Open File in Horizontal Split from File Explorer</h3> <C-x></code> while hovering on file in file explorer</p> Open File in Tab from File Explorer</h3> <C-t></code> while hovering on file in file explorer</p> Previous/Next Uncommitted Change in File Explorer</h3> Next: ]c</code></li>

Open File in Tab from File Explorer</h3> <C-t></code> while hovering on file in file explorer</p> Previous/Next Uncommitted Change in File Explorer</h3> Next: ]c</code></li>

Previous/Next Uncommitted Change in File Explorer</h3> Next: ]c</code></li>

Move Up a Directory in File Explorer</h3> -</code></p>

goto definition</h3> gd</code> from normal mode while hovering on symbol</p>

Fuzzy Search Files by Name</h3> <leader>f</code></p>

Populate quickfix window with ripgrep search results</h3> \</code> from normal mode</p>

vim-rhubarb</h2> Didn't know this existed but now can't live without it:</p> Open current buffer on Github.com in browser</h3> :GBrowse</code></p> Living Doc</h2> My intention is to update this doc as I learn more shortcuts worth remembering.</p>

Open current buffer on Github.com in browser</h3> :GBrowse</code></p>

Living Doc</h2> My intention is to update this doc as I learn more shortcuts worth remembering.</p>

Quick Notes on SQLite Capabilities

Drawback - Potential Data Loss</h3> Because you have to store your stateful database on the same "physical" server (really this could be bare metal, VM, or a container) as your application, if anything happens to your "physical server" then you also lose all the data too.</p>

Conclusion</h1> What tools are you using to help you live with SQLite?</p>

Backup Options for Open Source SQL Databases

Fun with Histograms - Prometheus Basics

Java LSP for Neovim - Java Basics

Simple Logging Façade for Java (slf4j) - Java Basics

Farewell to Mail-In-A-Box

Next Steps</h2> My plan is to write up my steps towards making this cluster a reality! Next stop, get postfix working.</p>

Review of Diagrams-as-Code for mdBook

Returning Trait Objects

Failure 1: return impl Trait</h2> I mistakenly thought impl Trait in return position would be the way to handle this case. Here's code that does NOT</strong> compile:</p>

Tracking Personal Finances through Email

Looking at Alternatives</h2> There's loads of options out there. I only looked at open source options that let me own the data. That eliminated things like LunchMoney, YooNeedABudget, and other paid options.</p>

Ledger & Friends Pros:</h4> Double-entry accounting</li> No 3rd parties</li> Portable data format</li> Mature technology</li> Excellent documentation</li> Active community</li>

Ledger & Friends Cons:</h4> Can't pull data from my bank / credit cards -- manual step here</li> No real time alerts</li> Single device / terminal or desktop website UIs</li> Limited categorization functionality</li>

GnuCash Pros</h4> Private - No Third Parties</li> Double Entry Accounting</li> Active development and community</li> Strong support for Quicken exports</li> Feature rich</li> Desktop GUI (and Android too)</li> Multi-currency</li>

GnuCash Cons</h4> Basically the same as ledger & friends...</p>

Summary of Alternatives</h3> No alternatives had these three features at the same time:</p> Automatic download of transactions</li> No third party data sharing</li> Multi-device</li>

Learnings</h3> Client side JavaScript could probably do a lot of stuff I hadn't consider before</li>

Learnings</h3> My bank's website might be slightly harder to hack than I first thought -- go bank!</li>

Learnings</h3> I needed to be more surgical in both planning and writing code</li>

Not Quite Success</h5> So It turns out, in rare cases there really are multiple transactions on the same day with the same description and amount and account. Gah! How do you distinguish between these cases?</p> You can't. Not using transaction emails.</p>

Web Analytics without Google - A Just Tracking Theory

Be the Friendly Store Keeper You Want To See in the World</h2> How can an individual site owner ensure their tracking is more like a friendly store keeper, and less like consumer espionage? Just don't use Google Analytics, and you'll stop feeding the corporate surveillance machine.</p>

Please Don't Deply Before You Test

Xeyes</h2>
We'll use the xeyes application as a simple test that things are working. A docker image exists for this at `fr3nd/xeyes</code>.</p>`

Helpful Source Material</h2>

Nassos Michas post</a></li>
This stackoverflow post</a> concerning permissions</li>
This very specific stackoverflow post</a> concerning eliminating terminals from showing up</li> </ul>

Move Up a Directory in File Explorer</h3>
`-</code></p>`

goto definition</h3>
`gd</code> from normal mode while hovering on symbol</p>`

Fuzzy Search Files by Name</h3>
`<leader>f</code></p>`

Populate quickfix window with ripgrep search results</h3>
`\</code> from normal mode</p>`

`vim-rhubarb</h2> Didn't know this existed but now can't live without it:</p> Open current buffer on Github.com in browser</h3> :GBrowse</code></p> Living Doc</h2> My intention is to update this doc as I learn more shortcuts worth remembering.</p>`

Open current buffer on Github.com in browser</h3>
`:GBrowse</code></p>`

`Living Doc</h2> My intention is to update this doc as I learn more shortcuts worth remembering.</p>`

Drawback - Potential Data Loss</h3>
Because you have to store your stateful database on the same "physical" server (really this could be bare metal, VM, or a container) as your application, if anything happens to your "physical server" then you also lose all the data too.</p>

Conclusion</h1>
What tools are you using to help you live with SQLite?</p>

Next Steps</h2>
My plan is to write up my steps towards making this cluster a reality! Next stop, get postfix working.</p>

Looking at Alternatives</h2>
There's loads of options out there. I only looked at open source options that let me own the data. That eliminated things like LunchMoney, YooNeedABudget, and other paid options.</p>

GnuCash Cons</h4>
Basically the same as ledger & friends...</p>

Not Quite Success</h5>
So It turns out, in rare cases there really are multiple transactions on the same day with the same description and amount and account. Gah! How do you distinguish between these cases?</p>
You can't. Not using transaction emails.</p>