How to Write Second Generation AppEngine Apps with Go 1.11

By Brian Muramatsu

2019-01-07

Overview

What is the ponzi2.io web site written in?

Go 1.11 with just the standard library! It also runs for FREE on the Second Generation AppEngine runtime without any AppEngine dependencies!^* Since the new runtime allows any extension or library, I decided to write a server from scratch inspired by suckless.org. In other words, I wanted to drop static content into folders, serve them at URLs matching those folder paths, and render shared headers and footers using the template package. Let's check out the full ~200 lines!^**

^* You still need an AppEngine app.yaml file. :-| Also, you might get charged a literal penny or two for Cloud Storage which is needed to compile and deploy your app.

^** The 200 lines does not include HTML files nor the GitHub and GitLab integration code. ;-)

Bursting the Bubble: app.yaml Lives!

Although the Go code has no dependencies on AppEngine libraries, you will still need an app.yaml file to eventually deploy it on AppEngine. Meanwhile, you can still write and run your server locally as if it were a regular Go program without an app.yaml. Let's get it out of the way though...

# https://cloud.google.com/appengine/docs/standard/go111/config/appref

runtime: go111

env_variables:
    PAYPAL_ENVIRONMENT: production
    CACHE_TEMPLATES: true

handlers:
- url: /.*
    script: auto
    secure: always
    redirect_http_response_code: 301

Notes:

The runtime directive is required and specifies the AppEngine runtime.
env_variables are optional. I defined a couple custom variables to trigger different behaviors on AppEngine versus on my local workstation for testing.
The handlers block is optional too, but I added this code to redirect all HTTP traffic to HTTPS. After all, we can't ask for donations on an unsecured site right? ;-)

Live Free with Environment Variables

Before diving into main, let's check out how to use environment variables to stay AppEngine-free. In this snippet, we read in a value from an environment variable to enable or disable caching templates, a custom feature specific to ponzi2.io. What this feature does is not important. What's interesting is that we use os.Getenv to read in the environment variable and have our code act on it. When testing locally, we start the server as a normal Go program which doesn't read from app.yaml. On the other hand, when we run our app on AppEngine, AppEngine will the start the server with the app.yaml and use the values defined there. In other words, environment variables allow us to remain independent from AppEngine yet they allow us to control our server's behavior when it does run on AppEngine.

var cacheTemplates = mustCacheTemplates()

func mustCacheTemplates() bool {
    v := strings.ToLower(os.Getenv("CACHE_TEMPLATES")) // YAML converts "true" to "True".
    switch v {
    case "", "false":
        return false
    case "true":
        return true
    default:
        log.Fatalf("bad cache templates: %s, want true or false", v)
    }
    return false
}

Notes:

Use os.Getenv to read in the environment variable's value.
We could have initialized these variables in main, but I think it's cleaner to define and initialize these at the top of the file. Up to you.

Getting Down to Business with main

OK! Now, in main, we use the standard library's net/http package to route URLs to handler functions and then launch the server on a port.

func main() {
    ctx := context.Background()

    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }

    log.Printf("cacheTemplates: %t", cacheTemplates)

    http.HandleFunc("/", templateFSHandler)
    http.HandleFunc("/favicon.ico", faviconHandler)
    http.Handle("/static/", http.StripPrefix("/static/", http.FileServer(http.Dir("static"))))

    log.Printf("serving on %s", port)
    log.Fatal(http.ListenAndServe(fmt.Sprintf(":%s", port), nil))
}

func faviconHandler(w http.ResponseWriter, r *http.Request) {
    http.ServeFile(w, r, "static/favicon.ico")
}

Notes:

Use os.Getenv to read in the port. PORT is a well-defined environment variable that AppEngine knows about.
/ is handled by the templateFSHandler function. It's kind of complicated, so we'll talk about it in the next section.
/favicon.ico is handled by faviconHandler function. It only needs to serve the single icon file, so it uses http.ServeFile to load that that single file and write it to the HTTP response.
Finally, a single line using http.StripPrefix, http.FileServer, and http.Dir handles all the other requests for static resources like images and CSS files. Here's an example of how a request for "/static/base.css" might work:
1. http.StripPrefix strips "/static/" off of "/static/base.css" yielding "base.css".
2. http.FileServer receives the request from http.StripPrefix and passes it to http.Dir.
3. http.Dir looks for "base.css" in the directory "static" and finds the file.

The Meat and Potatoes: Template File System Handler

templateFSHandler is the handler that allows me to easily create HTML files, drop them into a folder, and start serving them at a corresponding URL. It maps request URLs to templates on the server's file system and executes them. No code modifications needed! Let's check out the helper function, requestedTemplate, which does the mapping part before templateFSHandler calls Execute on it.

func requestedTemplate(r *http.Request) (*template.Template, error) {
    filepath := "templates" + r.URL.Path
    if strings.HasSuffix(filepath, "/") {
        filepath = path.Join(filepath, "index.html")
    }

    if cacheTemplates {
        if t, ok := filepath2Templates[filepath]; ok {
            return t, nil
        }
    }

    t, err := template.ParseFiles(filepath, "templates/base.html")
    if err != nil {
        return nil, err
    }

    if cacheTemplates {
        filepath2Templates[filepath] = t
        log.Printf("cached template: %s, num cached: %d", filepath, len(filepath2Templates))
    }

    return t, nil
}

Notes:

First, we takes the request path like "/tutorials/" and prepend "templates" to it to get "templates/tutorials/". This is the first step in translating the URL to a template on the file system.
Next, we add "index.html" to the path if the path ends with "/". So "templates/tutorials/" becomes "templates/tutorials/index.html". If the path was "templates/tutorials/appengine.html", we would not add "index.html", so the path would remain "templates/tutorials/appengine.html".
Finally, we call template.ParseFiles with that path and also include "templates/base.html". The base.html includes common templates like the header and footer that can be referenced by the template we just loaded.

Finish Line: Running Locally and Deploying to AppEngine!

Let's run the app locally and deploy it on AppEngine. First, here is the folder structure and files on my local workstation:

static

base.css
favicon.ico
pzchart.png
pzicon.png

templates

news

index.html

tutorials

appengine.html
graphql.html
index.html

base.html
index.html

app.yaml
main.go

Next, here's the command to run things locally. I run it in the same directory as main.go:

go generate ./... && go build -v -x && ./APP_NAME

Finally, here is what I do to deploy to AppEngine... and it works!

gcloud app deploy

In conclusion, this is just one example of how to write a simple static site on the new Second Generation AppEngine Runtime using Go 1.11. You can do whatever you write like maybe storing content in a DB... Oh, the possibilities are endless! Good luck!

Full Code

app.yaml

# https://cloud.google.com/appengine/docs/standard/go111/config/appref

runtime: go111

env_variables:
    PAYPAL_ENVIRONMENT: production
    CACHE_TEMPLATES: true

handlers:
- url: /.*
    script: auto
    secure: always
    redirect_http_response_code: 301

main.go

// This server can run on App Engine.
package main

import (
    "context"
    "fmt"
    "log"
    "net/http"
    "os"
    "path"
    "strings"
    "text/template"
    "time"

    "gitlab.com/btmura/kittyhawk/github"
    "gitlab.com/btmura/kittyhawk/gitlab"
)

var paypalEnvironment = mustPaypalEnvironment()

// mustPaypalEnvironment returns a valid Paypal environment or dies.
// This is used to fail early at startup, because getting paid is important!
func mustPaypalEnvironment() string {
    v := strings.ToLower(os.Getenv("PAYPAL_ENVIRONMENT"))
    switch v {
    case "", "sandbox":
        return "sandbox"
    case "production":
        return "production"
    default:
        log.Fatalf("bad paypal environment: %s, want sandbox or production", v)
    }
    return ""
}

var cacheTemplates = mustCacheTemplates()

func mustCacheTemplates() bool {
    v := strings.ToLower(os.Getenv("CACHE_TEMPLATES")) // YAML converts "true" to "True".
    switch v {
    case "", "false":
        return false
    case "true":
        return true
    default:
        log.Fatalf("bad cache templates: %s, want true or false", v)
    }
    return false
}

var (
    cachedPrivateCommitCount *int
    cachedPublicCommitCount  *int
)

var (
    privateCommitCount = gitlab.CommitCount
    publicCommitCount  = github.CommitCount
)

var filepath2Templates = map[string]*template.Template{}

func main() {
    ctx := context.Background()

    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }

    log.Printf("paypalEnvironment: %s", paypalEnvironment)
    log.Printf("cacheTemplates: %t", cacheTemplates)

    t := time.NewTicker(1 * time.Hour)
    defer t.Stop()
    go func() {
        refresh := func() {
            privc, err := privateCommitCount(ctx)
            if err != nil {
                log.Printf("privateCommitCount: %v", err)
                return
            }
            cachedPrivateCommitCount = &privc

            pubc, err := publicCommitCount(ctx)
            if err != nil {
                log.Printf("publicCommitCount: %v", err)
                return
            }
            cachedPublicCommitCount = &pubc

            log.Printf("private: %d, public: %d", *cachedPrivateCommitCount, *cachedPublicCommitCount)
        }

        refresh()

        for t := range t.C {
            log.Printf("refreshing commit counts at %v", t)
            refresh()
        }
    }()

    http.HandleFunc("/", templateFSHandler)
    http.HandleFunc("/favicon.ico", faviconHandler)
    http.Handle("/static/", http.StripPrefix("/static/", http.FileServer(http.Dir("static"))))

    log.Printf("serving on %s", port)
    log.Fatal(http.ListenAndServe(fmt.Sprintf(":%s", port), nil))
}

func templateFSHandler(w http.ResponseWriter, r *http.Request) {
    handleErr := func(tag string, err error) {
        var code int
        switch {
        case os.IsNotExist(err):
            code = http.StatusNotFound
        default:
            code = http.StatusInternalServerError
        }
        log.Printf("[%d] %s: %s", code, tag, err)
        w.WriteHeader(code)
    }

    t, err := requestedTemplate(r)
    if err != nil {
        handleErr("requestTemplate", err)
        return
    }

    if cachedPrivateCommitCount == nil {
        c, err := privateCommitCount(r.Context())
        if err != nil {
            handleErr("privateCommitCount", err)
            return
        }
        cachedPrivateCommitCount = &c
    }

    if cachedPublicCommitCount == nil {
        c, err := publicCommitCount(r.Context())
        if err != nil {
            handleErr("publicCommitCount", err)
            return
        }
        cachedPublicCommitCount = &c
    }

    // Don't show a negative diff count...
    diff := *cachedPrivateCommitCount - *cachedPublicCommitCount
    if diff < 0 {
        diff = 0
    }

    args := struct {
        PrivateCommitCount int
        PublicCommitCount  int
        CommitCountDiff    int
        PaypalEnvironment  string
    }{
        PrivateCommitCount: *cachedPrivateCommitCount,
        PublicCommitCount:  *cachedPublicCommitCount,
        CommitCountDiff:    diff,
        PaypalEnvironment:  paypalEnvironment,
    }

    if err := t.Execute(w, args); err != nil {
        handleErr("t.Execute", err)
        return
    }
}

func requestedTemplate(r *http.Request) (*template.Template, error) {
    filepath := "templates" + r.URL.Path
    if strings.HasSuffix(filepath, "/") {
        filepath = path.Join(filepath, "index.html")
    }

    if cacheTemplates {
        if t, ok := filepath2Templates[filepath]; ok {
            return t, nil
        }
    }

    t, err := template.ParseFiles(filepath, "templates/base.html")
    if err != nil {
        return nil, err
    }

    if cacheTemplates {
        filepath2Templates[filepath] = t
        log.Printf("cached template: %s, num cached: %d", filepath, len(filepath2Templates))
    }

    return t, nil
}

func faviconHandler(w http.ResponseWriter, r *http.Request) {
    http.ServeFile(w, r, "static/favicon.ico")
}

Support

Donate to keep development chugging (0 commits) or just star the repository like 0 others!

bitcoin:38vo2oWYmqBUXCxL3avpueye6dPRahX7gC