Introduction
Software performance is critical to a SaaS company like Stream and while the majority of our infrastructure is written in Python, we are actively porting portions of our code base to Go.
Stream is an API for building scalable feeds and now handles over 20 billion feed updates a month. While Python has been fast enough for many things, the additional speed and efficiency of Go is becoming important to us. (If you’ve never tried Stream, here’s a 5 minute interactive tutorial)
This migration from other scripting languages like Python to Go is becoming pretty commonplace in our industry. Before Stream I worked as a lead engineer at SendGrid for 4 years and the last 18 months was spent replacing sluggish code from Ruby and Python, and writing internal and external APIs, in more performant Go.
This tutorial will teach you how to build a RESTful API using Go and Cassandra. Cassandra is a NoSQL database that automatically shards your data across a cluster of machines. Here at Stream we use Cassandra for the storage of news feeds and activity streams. it’s performance, linear scalability and reliability make it the #1 datastore for many large platforms such as Netflix, GitHub, Reddit, Instagram, Sony, Coursera, Spotify, and eBay.
The source code for this tutorial can be found on our GitHub repository.
Step 1 – Cassandra
By far, the easiest way to get Cassandra installed and running is to use the Cassandra Cluster Management script, and we’ll use homebrew for setting up tooling on a Mac; setup on Linux should be very similar but your mileage may vary on Windows platforms. Running “brew install ccm” will get you started, and we’ll also need a Java environment, which is bundled in the “maven” package.
A. Cassandra Setup
$ brew install ccm maven
To query Cassandra from the command line we need a tool called cqlsh. You can easily install the cqlsh command with pip:
$ pip install cqlsh==5.0.3
To begin, we need to install Cassandra and create a cluster. The following command downloads and install Cassandra v3.9 and creates a cluster named streamdemoapi
$ ccm create -v 3.9 streamdemoapi
Next we need to add one or more nodes to our cluster. For the tutorial we only need a single node:
$ ccm populate -n 1 $ ccm start
You can verify the cluster booted correctly by running ccm status.
$ ccm status
Finally we create a keyspace with single-node replication. If we had set up multiple nodes in the previous step, we could specify how many nodes we wanted to replicate data onto.
$ echo "CREATE KEYSPACE streamdemoapi WITH \
replication = {'class': 'SimpleStrategy', 'replication_factor' : 1};" | cqlsh --cqlversion 3.4.2
Congratulations, you now have a Cassandra cluster + keyspace up and running.
Note: For a production Cassandra setup there are more things to keep in mind: We recommend starting at Datastax and browsing their tutorial series on using Cassandra.
B. Cassandra Schema
We will define two tables to hold our data for users and messages. We recommend using UUIDs for unique resource identifiers when sending an activity to Stream, so we’ve used UUIDs as our primary key in our table structures to identify each message and user in our system.
$ echo " use streamdemoapi; create table messages ( id UUID, user_id UUID, Message text, PRIMARY KEY(id) );" | cqlsh --cqlversion 3.4.2
Our user table will contain some simple data about each person using our application, including their first name, last name, age, city and email address.
$ echo " use streamdemoapi; CREATE TABLE users ( id UUID, firstname text, lastname text, age int, email text, city text, PRIMARY KEY (id) );" | cqlsh --cqlversion 3.4.2
Step 2 – Go
Installing Go on a Mac is super easy with homebrew. Installation instructions for Linux and Windows can be found at golang.org. The current version as of this writing is Go 1.7.4.
A. Go setup
Let’s install the binaries and standard library for Go on our Mac:
$ brew install go
With the binary installed, we will need to configure the “GOPATH” folder where we’ll install third-party Go libraries and packages. Within your terminal shell configuration (for example, .bashrc or .zshrc) we will add some new environment settings:
export GOPATH=$HOME/golang export GOROOT=/usr/local/opt/go/libexec
The GOROOT setting tells our shell where to find Go binaries and the standard libraries. This may not be required on your system if you’re using Go 1.7 or better, as it can find the path on its own if the main “go” binary itself in your PATH.
Restart your shell, or source your shell configuration file, and then create this new path:
$ mkdir $HOME/golang
B. Go Dependencies
Go can be very peculiar about dependency management and the Golang team recommends “vendoring” your dependencies within the same folder as your project source code to ensure you don’t have to worry about running “go get” in the future and ending up with a newer version of a library than your code can utilize. Our tutorial will install all packages within our GOPATH for simplicity.
With our GOPATH defined above, we can install third-party Go packages, such as the Cassandra client, by running the “go get” command followed by the origin of the package. To install the Cassandra driver for our project, we’ll issue the following command:
$ go get github.com/gocql/gocql
You will find the new package installed under your home folder, in the “golang” folder we created above, in a subfolder called “src”. In that src folder, you’ll see another folder for github.com and so on for each of the packages you install.
For our project we’ll also need the following libraries which we’ll describe later in this tutorial:
$ go get github.com/gorilla/mux $ go get github.com/GetStream/stream-go
If you are creating a Go application that will not have subpackages, you can develop the code anywhere on your filesystem. However, since this application (and most other apps you will develop) will use subpackages, your code must live in your GOPATH folder.
To make sure you’re in the correct folder to begin your code, navigate to your GOPATH/src folder with this command:
$ cd $GOPATH/src
Since you will have your code in a repository somewhere, you will place your project code in a folder based on your username and project name. For example, if your username on GitHub was johnsmith and your project was awesomeproject, then you would create your project under the following path:
$ mkdir $GOPATH/src/github.com/johnsmith/awesomeproject $ cd $GOPATH/src/github.com/johnsmith/awesomeproject
Technically, it doesn’t matter what this path is, but this is a common standard employed by other Go developers, so it will cause fewer problems if you conform to this standard.
At this point, you can start writing your code!
Step 3 – Your first REST endpoint
When we’re all finished with this tutorial, our API contract will look something like this:
GET / — heartbeat check if our API is online
GET /users — fetch all users from the database
GET /users/UUID — fetch an individual user from the database
POST /users/new — create a new userGET /messages — fetch all messages from Stream (with the database as a backup)
GET /messages/UUID — fetch an individual message from the database
POST /messages/new — create a new message
One big question faced by a lot of Go API developers is whether to use a third-party web framework or to use the standard libraries which ship with Go. In my experience building APIs with both standard libraries and third-party frameworks, I think it’s really a personal call. Certainly, rewriting advanced routing rules and dispatch/middleware handlers every time you create an application with the standard libraries is time-consuming, but using third-party frameworks can often place constraints on your code where you must do things “their” way. The standard library is certainly more than capable of managing a simple API setup, but sometimes it’s nice to have well-tested third-party code that make it easier and quicker to get started.
In our example application, we opted to use the popular gorilla/mux project to build our API.
Let’s start with the heartbeat endpoint.
Go #protip: Good project structure in Go means as little logic in your applications’ /main.go starting point, and breaking down other logic/branches into subfolders to separate your code’s concerns, so code we write after this part of the tutorial will all go into subfolders, which creates subpackages.
The first line of your Go script is the name of your package, and Go will always expect the root level of your project to be a package called “main”. We must tell our Go script which other libraries we intend to use with the “import” command. To get started with gorilla/mux and a basic API endpoint for a status check (also called a “heartbeat”). Create a file called main.go at the base of your project folder (ie, johnsmith/awesomeproject/main.go) and add the following code:
package main import ( "net/http" "log" "encoding/json" "github.com/gorilla/mux" )
In order for gorilla/mux to return any payload, we must define the exact structure of the data. In our example we will send back a string for “status” and an integer for a status code. For example, if our API is running successfully, we could send back a status of “OK” and a code of 200, which is a typical HTTP response for a successful web connection.
type heartbeatResponse struct {
Status string `json:"status"`
Code int `json:"code"`
}
The `json:”status”` portion of the field definition above will tell our JSON-encoder to rename the “Status” and “Code” fields during the encoding process to their lowercase versions.
Next, we define our “main” function, which is a reserved name in Go for where your code will begin. Within our main function, we need to set up a new gorilla/mux router to handle our API endpoints according to the contract we described above. We can add a new API route after our router is declared, and give it the name of another function to execute when a request is made for that endpoint. In this case, we’ll call a new function called heartbeat:
func main() {
router := mux.NewRouter().StrictSlash(true)
router.HandleFunc("/", heartbeat)
log.Fatal(http.ListenAndServe(":8080", router))
}
The last line of code tells Go to listen on port 8080, which will also log any errors returned and throw a fatal error to crash our application.
Finally, we must declare our heartbeat() function for the “/” endpoint, so we’ll add code this after our main() function:
func heartbeat(w http.ResponseWriter, r *http.Request) {
json.NewEncoder(w).Encode(heartbeatResponse{Status: "OK", Code: 200})
}
The handler code inside heartbeat() tells gorilla/mux to use a JSON encoder on our output handle (our http.ResponseWriter variable simply called “w”), and encode our data structure that we defined above our main() function which we called heartbeatResponse, and to set our “OK” status string, and our status code of 200, indicating our API is running properly.
Now we save our code, and we can execute it in our terminal by running the following command:
$ go run main.go
If there are no errors, you won’t see any output, and the program will appear to “hang” but it’s actually running in the background, listening on port 8080. To test this, open your browser and go to the following address:
http://localhost:8080/
If all goes well, you should see some JSON text in your browser which says:
{“status”: “OK”, “code”: 200}
To stop the API from running, you can press CTRL-C to kill the Go application.
Congratulations — you’ve created a Go-powered API! It’s not terribly useful yet, though, so we can continue adding code, such as the remaining routes for our API contract for getting or creating users and messages. In our final /main.go code, linked below, we’ll also connect to our local Cassandra instance and verify our Stream credentials allow us to connect.
Final Source: /main.go
Step 4 – Using Cassandra in Go
init()will have all of those init() functions called immediately before the process’main()function. The caution here is that if you have several submodules with init() functions declared, there is no guarantee of run order so you must ensure you don’t inadvertently impose a required order by having mixed dependencies. If you must have these initializations happen in a specific order, it’s best to declare an exported method in each submodule and call them explicitly from your process’ main() function.Because the Cassandra code will live in a subfolder called Cassandra, it becomes a subpackage to our primary Go code, and therefore must have a new package name, declared on the first line of our code. Next, we will import the GoCQL library package. (we also import the “fmt” standard library to print a message when we are connected to Cassandra)
Create a file Cassandra/main.go and add this code:
package Cassandra import ( "github.com/gocql/gocql" "fmt" )
In order to use our Cassandra connection elsewhere in our code, we will declare a variable called Session. To use a subpackage variable in other code outside of this subfolder, we must name the variable with the first letter as an uppercase letter. A variable named “session” (all lowercase) will not be visible to other packages.
var Session *gocql.Session
Next, we will declare our init() function which will run as soon as our API code executes (but before our application’s /main.go main() function is run):
func init() {
var err error
cluster := gocql.NewCluster("127.0.0.1")
cluster.Keyspace = "streamdemoapi"
Session, err = cluster.CreateSession()
if err != nil {
panic(err)
}
fmt.Println("cassandra init done")
}
To connect to Cassandra, we use the GoSQL NewCluster() method and pass an IP address or hostname and store that connection in a variable called “cluster”. Next, we declare which cluster name we wish to use; in this case we define “streamdemoapi” as our cluster name. Finally, we create a session on the cluster and assign that to our exportable Session variable.
The GoCQL project makes interfacing with Go super easy, and many queries can be written just like common SQL for selecting or inserting data into a database, which we’ll show in the next submodule.
The last steps are to add one more import to our application’s /main.go (not in Cassandra/main.go):
package main import ( ... "github.com/username/projectname/Cassandra" ... )
And finally we tell our application’s /main.go main() function to defer closing our Cassandra connection:
func main() {
CassandraSession := Cassandra.Session
defer CassandraSession.Close()
router := mux.NewRouter().StrictSlash(true)
...
}
The first line will utilize our Cassandra session from our subpackage and store that in a new local variable. The defer line tells Go to disconnect from Cassandra if the application’s /main.go main() function exits for any reason.
You can verify everything worked by running:
$ go run main.go
And you should see a message in your terminal that the Cassandra initialization is done.
Final source: Cassandra/main.go
Step 5 – Creating and Reading Users
A. Creating Users
To create or fetch user data from Cassandra, we create a sub-package (and folder) called Users and have our data structures and logic split into separate files within that folder. All files within this folder must also have a matching package declaration that is different from all other subpackages in our application.
package Users
Create a file, Users/structs.go, to contain our data structures, and Users/processing.go for handling form data. You can refer to our GitHub source for the exact code, links are below, and we’ll describe these in detail later. To simplify getting started right now:
>> Copy this file to your project as Users/structs.go
>> Copy this file to your project as Users/processing.go
To add data to Cassandra, we will create a file called Users/post.go to handle our POST operation on our API.
In our imports, we will include a reference to our Cassandra subpackage
package Users import ( "net/http" "github.com/gocql/gocql" "encoding/json" "github.com/username/projectname/Cassandra" "fmt" )
We will make a function called Post() which take a request and response variable from gorilla/mux. Let’s explain what happens within the function:
func Post(w http.ResponseWriter, r *http.Request) {
var errs []string
var gocqlUuid gocql.UUID
// FormToUser() is included in Users/processing.go
// we will describe this later
user, errs := FormToUser(r)
// have we created a user correctly
var created bool = false
// if we had no errors from FormToUser, we will
// attempt to save our data to Cassandra
if len(errs) == 0 {
fmt.Println("creating a new user")
// generate a unique UUID for this user
gocqlUuid = gocql.TimeUUID()
// write data to Cassandra
if err := Cassandra.Session.Query(`
INSERT INTO users (id, firstname, lastname, email, city, age) VALUES (?, ?, ?, ?, ?, ?)`,
gocqlUuid, user.FirstName, user.LastName, user.Email, user.City, user.Age).Exec(); err != nil {
errs = append(errs, err.Error())
} else {
created = true
}
}
// depending on whether we created the user, return the
// resource ID in a JSON payload, or return our errors
if created {
fmt.Println("user_id", gocqlUuid)
json.NewEncoder(w).Encode(NewUserResponse{ID: gocqlUuid})
} else {
fmt.Println("errors", errs)
json.NewEncoder(w).Encode(ErrorResponse{Errors: errs})
}
}
Next, if our application’s main.go, we need to add the Users subpackage to our imports and add a new REST endpoint to our gorilla/mux router to call this Post() method:
import (
...
"github.com/username/projectname/Cassandra"
"github.com/username/projectname/Users"
...
)
func main() {
...
router := mux.NewRouter().StrictSlash(true)
router.HandleFunc("/", Heartbeat)
router.HandleFunc("/users/new", Users.Post)
...
}
Testing our new endpoint
>>> Curl example of posting to the User endpoint
curl -X POST \ -H "Content-Type: application/x-www-form-urlencoded" \ -d 'firstname=Ian&lastname=Douglas&city=Boulder&email=ian@getstream.io&age=42' \ "http://localhost:8080/users/new"
The response you get should look similar to:
{"id":"75a243c1-e356-11e6-81c2-c4b301bb0fa9"}
Now that we’ve tested that our POST operation works, let’s rewind and describe what Users/processing.go is actually doing.
At a high level, gorilla/mux provides a method on your request interface called PostFormValue which takes a string parameter and returns a string of data (or empty string if the parameter was not present in the form data). We check the string length for each parameter that we require and if the length is 0 we push an error onto an array and return those errors to the user. We cycle through all form variables we expect to find and also validate that the age parameter is an integer.
Let’s look at this closer:
Our Users/processing.go file is part of our Users subpackage, and imports a few standard libraries.
package Users import ( "net/http" "strconv" )
Next, we have a function which allows us to append a string onto an existing array of strings. Go does not have a built-in mechanism for this, so we wrote our own:
func appendError(errs []string, errStr string) ([]string) {
if len(errStr) > 0 {
errs = append(errs, errStr)
}
return errs
}
Now we’ll define a function which will take our request interface, and a string, and return the field data or an error in one of two return positions. As mentioned above, gorilla/mux has a PostFormValue() method defined for us to fetch form data from an HTTP request.
func processFormField(r *http.Request, field string) (string, string) {
fieldData := r.PostFormValue(field)
if len(fieldData) == 0 {
return "", "Missing '" + field + "' parameter, cannot continue"
}
return fieldData, ""
}
Finally, we have our code that cycles through each of the fields we deem to be “required” fields on our form data:
func FormToUser(r *http.Request) (User, []string) {
var user User
var errStr, ageStr string
var errs []string
var err error
user.FirstName, errStr = processFormField(r, "firstname")
errs = appendError(errs, errStr)
user.LastName, errStr = processFormField(r, "lastname")
errs = appendError(errs, errStr)
user.Email, errStr = processFormField(r, "email")
errs = appendError(errs, errStr)
user.City, errStr = processFormField(r, "city")
errs = appendError(errs, errStr)
ageStr, errStr = processFormField(r, "age")
if len(errStr) != 0 {
errs = append(errs, errStr)
} else {
user.Age, err = strconv.Atoi(ageStr)
if err != nil {
errs = append(errs, "Parameter 'age' not an integer")
}
}
return user, errs
}
B. Reading users
The Users/get.go script handles our two GET requests: Get() is called for fetching all users (http://localhost:8080/users), and GetOne() to fetch a single user with a resource ID (http://localhost:8080/users/UUID).
We start out by declaring our package name and our imports. Because we’ll be pulling data from Cassandra using the GoCQL library, we must make sure to include those. We also include the gorilla/mux library to access the UUID value from our GetOne() endpoint.
package Users import ( "net/http" "github.com/gocql/gocql" "encoding/json" "github.com/username/projectname/Cassandra" "github.com/gorilla/mux" "fmt" )
To begin, we’ll outline our code to fetch all users. A best practice on this, of course, would restrict the number of results per call and to take in limit/offset parameters, but we will skip that in this tutorial.
Querying Cassandra data using GoCQL looks just like SQL you would write for MySQL or PostgreSQL. We pass the query as a string to our session’s Cassandra.Query() function. The Query() function returns an iterable, and we use the iterable’s MapScan() function to copy data into a map[string]interface (we use a map[string]interface because perhaps not all data in the table is string data to use map[string]string).
int(myVariable)similar to C, Python, etc.. However, when you have data in a Go interface, you must use a suffix-style casting, which looks likemyVariable.(int)We append each User structure to an array as we iterate through each user. One quirk of the GoCQL iterable is that we must empty the map before the next call to MapScan(), which is the last step in our for loop. Finally, we package up our array of User structures and return it as JSON. Here’s the code:
func Get(w http.ResponseWriter, r *http.Request) {
var userList []User
m := map[string]interface{}{}
query := "SELECT id,age,firstname,lastname,city,email FROM users"
iterable := Cassandra.Session.Query(query).Iter()
for iterable.MapScan(m) {
userList = append(userList, User{
ID: m["id"].(gocql.UUID),
Age: m["age"].(int),
FirstName: m["firstname"].(string),
LastName: m["lastname"].(string),
Email: m["email"].(string),
City: m["city"].(string),
})
m = map[string]interface{}{}
}
json.NewEncoder(w).Encode(UsersResponse{Users: userList})
}
Our endpoint code in our application’s /main.go main() function will look like this, and be placed next to our other routes:
router.HandleFunc("/users", Users.Get)
Next, our code to fetch a single user by their UUID. Let’s start by showing you the endpoint route that goes in our application’s /main.go main() function:
router.HandleFunc("/users/{user_uuid}", Users.GetOne)
You’ll notice that unlike our other routes, we have a string wrapped in curly braces called user_uuid. This string declares a variable route name through which we can extract the user’s UUID from the requested URL.
To begin, we extract all URL variables (declared in those curly braces) into a map called vars, and then access user_uuid from that map. Next, we validate that string by passing it to to GoCQL’s parser for Cassandra UUID values to determine if the value is in fact a UUID. If the validation passes, we fetch that user’s data from Cassandra similar to how we fetched all users in the code above. The difference in this call is the ? placeholder in our query for binding parameters to avoid injection attacks. When we call the Cassandra session’s Query() method, we not only pass the string of the query, but each parameter to pass into the query to fill each question mark. In this example we also show how you can tell Cassandra to have single-node consistency when fetching data; this is, of course, redundant since we’re only running on a single node. We then iterate over our result set (even though we set a limit of one row) and return the User structure or an error.
func GetOne(w http.ResponseWriter, r *http.Request) {
var user User
var errs []string
var found bool = false
vars := mux.Vars(r)
id := vars["user_uuid"]
uuid, err := gocql.ParseUUID(id)
if err != nil {
errs = append(errs, err.Error())
} else {
m := map[string]interface{}{}
query := "SELECT id,age,firstname,lastname,city,email FROM users WHERE id=? LIMIT 1"
iterable := Cassandra.Session.Query(query, uuid).Consistency(gocql.One).Iter()
for iterable.MapScan(m) {
found = true
user = User{
ID: m["id"].(gocql.UUID),
Age: m["age"].(int),
FirstName: m["firstname"].(string),
LastName: m["lastname"].(string),
Email: m["email"].(string),
City: m["city"].(string),
}
}
if !found {
errs = append(errs, "User not found")
}
}
if found {
json.NewEncoder(w).Encode(GetUserResponse{User: user})
} else {
json.NewEncoder(w).Encode(ErrorResponse{Errors: errs})
}
}
Finally in our Users/get.go code, we have a function to fetch a user’s first and last name to “enrich” some data. We will use this function in our Messages code shortly, and so we must declare the function with a capital E so it is visible to other subpackages in our project.
As a best practice, enrichment code would usually retrieve an entire User structure not just a few fields, but we’re taking an easy approach on this for our example application. The Enrich() function takes an array of UUID values, does a single database lookup, and creates a map of the UUID (as a string, this is important later) and the user’s concatenated first and last name. If an empty list of UUIDs is passed, an empty map is returned, otherwise it will return any results it finds in Cassandra.
func Enrich(uuids []gocql.UUID) map[string]string {
if len(uuids) > 0 {
names := map[string]string{}
m := map[string]interface{}{}
query := "SELECT id,firstname,lastname FROM users WHERE id IN ?"
iterable := Cassandra.Session.Query(query, uuids).Iter()
for iterable.MapScan(m) {
fmt.Println("m", m)
user_id := m["id"].(gocql.UUID)
names[user_id.String()] = fmt.Sprintf("%s %s", m["firstname"].(string), m["lastname"].(string))
m = map[string]interface{}{}
}
return names
}
return map[string]string{}
}
Full Source: Users/*.go
Step 6 – Connecting to Stream
Great, we’ve learned how to set up the beginnings of our RESTful API with Go and Cassandra. As a next step let’s see how we can integrate Stream, our API for building scalable feeds and activity streams. We’ll use Stream to store a copy of our messages, so when our application goes viral it will be Stream’s job to handle the scalability of handling our activity feed.
Our Stream code will live in a separate subfolder as well, called “Stream” with its own main.go. Like the Cassandra code, it must have its own unique package name as well. We will import the Stream Go SDK library as well as the standard library for reporting errors. Inside Stream/main.go place the following code:
package Stream import ( getstream "github.com/GetStream/stream-go" "errors" )
Next, we will declare an exportable variable called Client which we will use elsewhere in our code:
var Client *getstream.Client
Unlike our Cassandra code, this subpackage will not use the init() method so that the Connect() method will be called from our API application’s /main.go main() function. It will validate that we’ve passed in our API credentials and a region for the Stream SDK client and return an error if any of those items are missing. Next, the Stream SDK will create a new connection and assign that to our Client variable, or return an error if the connection failed.
func Connect(apiKey string, apiSecret string, apiRegion string) error {
var err error
if apiKey == "" || apiSecret == "" || apiRegion == "" {
return errors.New("Config not complete")
}
Client, err = getstream.New(&getstream.Config{
APIKey: apiKey,
APISecret: apiSecret,
Location: apiRegion,
})
return err
}
Since we’re not using an init() call, we will require our application’s /main.go main() function to manually connect to Stream, and here’s what that code could look like:
func main() {
err := Stream.Connect(
"api_key_goes_here",
"api_secret_goes_here",
"us-east")
if err != nil {
log.Fatal("Could not connect to Stream, abort")
}
CassandraSession := Cassandra.Session
defer CassandraSession.Close()
router := mux.NewRouter().StrictSlash(true)
...
}
Final source: Stream/main.go
Step 8 – Messages
We won’t go into the code for our messages as deeply as users because much of the structure is the same. Of course, all of our Messages scripts need to be declared with another unique package name:
package Messages
Our imports will also look the same, importing some standard libraries, gorilla/mux, and our Cassandra code. The GET and POST operations will also import our Stream submodules (not all scripts will need both):
import ( ... "github.com/username/projectname/Stream" getstream "github.com/GetStream/stream-go" ... )
Go #protip: Adding a word in front of an import like the Stream Go SDK above (“getstream”) will allow you to alias the subpackage namespace. In our tutorial, our Stream code uses a package name of “Stream” which conflicts with the SDK package, so we can use “getstream” as an alias when we reference the SDK calls.
A. Creating Messages
Like our Users code, we have a Messages/structs.go script which declares our data structures. Our first structure holds the message details from Cassandra, plus one extra field for the user’s full name (more on this later). The next two structures are for requesting one or all messages.
The processing.go file is a much simpler version of the user’s processing script and handles incoming form data for the POST request to save a new message. It only declares whether a required field is missing in the form data.
Before we get into the code, let’s review at a high level what our Message/post.go script will do for us, because creating a new message becomes more complex than just creating a user. First, we must validate our data points and save it in Cassandra, but then we’ll also forward the activity to Stream as an activity on a flat feed called “messages” to retrieve later.
package Messages import ( "net/http" "github.com/gocql/gocql" "encoding/json" "github.com/username/projectname/Stream" "github.com/username/projectname/Cassandra" getstream "github.com/GetStream/stream-go" )
Our message POST will require a user’s UUID to be part of our form data, as well as the text of the message itself. The validations below will check that the user_id passed to our code is a UUID, and that the message text has a length greater than 0. Of course, for simplicity, we skip validating that the user_id UUID is valid, and we also skip validating a maximum length on our message or perhaps other techniques to detect spam or unwanted behavior.
func Post(w http.ResponseWriter, r *http.Request) {
var errs []string
var errStr, userIdStr, message string
if userIdStr, errStr = processFormField(r, "user_id"); len(errStr) != 0 {
errs = append(errs, errStr)
}
user_id, err := gocql.ParseUUID(userIdStr)
if err != nil {
errs = append(errs, "Parameter 'user_id' not a UUID")
}
if message, errStr = processFormField(r, "message"); len(errStr) != 0 {
errs = append(errs, errStr)
}
Once our validation is proven, we generate a unique UUID for our message and insert the message into Cassandra similar to how we inserted our user record.
gocqlUuid := gocql.TimeUUID()
var created bool = false
if len(errs) == 0 {
if err := Cassandra.Session.Query(`
INSERT INTO messages (id, user_id, message) VALUES (?, ?, ?)`,
gocqlUuid, user_id, message).Exec(); err != nil {
errs = append(errs, err.Error())
} else {
created = true
}
}
If our message was created successfully in Cassandra, we can now build an Activity and send it to Stream. We start out by identifying our flat feed group called “messages” and passing an identifier as the word “global”; as mentioned earlier, we recommend passing a UUID value to avoid collisions, but this is common for global feeds of data that you want all users to see. We build the feed reference by calling our exported Client variable in our Stream subpackage, and calling the .FlatFeed() method.
if created {
// send message to Stream
globalMessages, err := Stream.Client.FlatFeed("messages", "global")
With our feed in place, we now create an Activity structure and pass it to Stream. At the very minimum, the Stream API requires three pieces of data to build an activity: an Actor, a Verb, and an Object. The actor is usually set to the user who is creating the activity. The verb can be any string up to 20 characters in length, and we’ll set that to “post” for users posting a message. The object reference in this case is the UUID of the message we just saved in Cassandra.
The Stream API also allows metadata to be passed in activities, and the total payload of a message can be about 1kb of key/value pairs of string data. Normally, your application would use the Object ID described above to fetch the message structure, but on an application like this, the messages table could be quite large. So, to save us a database lookup later on our messages table, we will assume that messages sent to our API are short (say, Twitter-length in size), and store the text of the message in the activity metadata. This way, when we retrieve the feed from Stream later, we have less database activity.
if err == nil {
globalMessages.AddActivity(&getstream.Activity{
Actor: getstream.FeedID(user_id.String()),
Verb: "post",
Object: getstream.FeedID(gocqlUuid.String()),
MetaData: map[string]string{
// add as many custom keys/values here as you like
"message": message,
},
})
}
json.NewEncoder(w).Encode(NewMessageResponse{ID: gocqlUuid})
} else {
json.NewEncoder(w).Encode(ErrorResponse{Errors: errs})
}
}
B. Reading Messages
Our Messages/get.go script will also be more complex than our Users code. Fetching a single message will pull it from the database but fetching all messages will first attempt to pull the messages from Stream; if Stream fails, we will look them up in the database.
package Messages import ( "net/http" "github.com/gocql/gocql" "encoding/json" "github.com/gorilla/mux" "fmt" "github.com/username/projectname/Stream" "github.com/username/projectname/Cassandra" "github.com/username/projectname/Users" )
Our GetOne() method will be called only when our API is fetching a single message. It will use the gorilla/mux Vars() call again (like we did with users) to parse a value out of our URL string, in this case something called {message_uuid}. We validate that the value is indeed a UUID, and then fetch that from our database.
Our GetOne() and Get() calls will also “enrich” some data from the messages. This is typical behavior with applications using Stream since our Activity payload is relatively small. Most of the time, your application will only send us references to users, messages, posts, and other object types, which are then retrieved in an efficient manner from your database. In our example, we will take the user_id UUID from our message and turn that into a string of just the user’s first and last name; more typically, your application would pull the entire user record to access any of the fields.
We’ve included our Users subpackage in the imports above so we have access to the Enrich() code in the Users/get.go script. This method takes a list of user_id UUIDs and returns a map of firstname/lastname strings.
func GetOne(w http.ResponseWriter, r *http.Request) {
var message Message
var errs []string
var found bool = false
vars := mux.Vars(r)
id := vars["message_uuid"]
uuid, err := gocql.ParseUUID(id)
if err != nil {
errs = append(errs, err.Error())
} else {
m := map[string]interface{}{}
query := "SELECT id,user_id,message FROM messages WHERE id=? LIMIT 1"
iterable := Cassandra.Session.Query(query, uuid).Consistency(gocql.One).Iter()
for iterable.MapScan(m) {
found = true
user_id := m["user_id"].(gocql.UUID)
names := Users.Enrich([]gocql.UUID{user_id})
fmt.Println("names", names)
message = Message{
ID: user_id,
UserID: m["user_id"].(gocql.UUID),
UserFullName: names[user_id.String()],
Message: m["message"].(string),
}
}
if !found {
errs = append(errs, "Message not found")
}
}
if found {
json.NewEncoder(w).Encode(GetMessageResponse{Message: message})
} else {
json.NewEncoder(w).Encode(ErrorResponse{Errors: errs})
}
}
Finally, our Get() method will fetch all messages in our “messages” feed at Stream. If something goes wrong, we need a backup and so we’ll retrieve the messages from Cassandra. Either way, we’ll strip out all user_id UUIDs while we iterate over the data, fetch their firstname/lastname strings, and enrich the message data to include the author’s name in our outgoing payload.
func Get(w http.ResponseWriter, r *http.Request) {
var messageList []Message
var enrichedMessages []Message
var userList []gocql.UUID
var err error
m := map[string]interface{}{}
Next, we create our reference to our flat feed as we did in Messages/post.go using our exported Client variable from our Stream subpackage.
globalMessages, err := Stream.Client.FlatFeed("messages", "global")
Now we’ll attempt to fetch our flat feed from Stream by calling the .Activities() method on our feed reference, and print a message saying that we have successfully fetched the feed. Then we iterate over the list of activities, and extract all of the user_id UUIDs.
// fetch from Stream
if err == nil {
activities, err := globalMessages.Activities(nil)
if err == nil {
fmt.Println("Fetching activities from Stream")
for _, activity := range activities.Activities {
fmt.Println(activity)
user_id, _ := gocql.ParseUUID(activity.Actor.Value())
message_id, _ := gocql.ParseUUID(activity.Object.Value())
messageList = append(messageList, Message{
ID: message_id,
UserID: user_id,
Message: activity.MetaData["message"],
})
userList = append(userList, user_id)
}
}
}
However, this is the internet we’re talking about and sometimes things fail. We could build in a retry mechanism with exponential back-off, but our users want to see messages as soon as possible, so let’s fetch them from Cassandra as our backup plan. While we iterate over those messages, we’ll also pull out the user_id UUIDs into a list.
// if Stream fails, pull from database instead
if err != nil {
fmt.Println("Fetching activities from Database")
query := "SELECT id,user_id,message FROM messages"
iterable := Cassandra.Session.Query(query).Iter()
for iterable.MapScan(m) {
user_id := m["user_id"].(gocql.UUID)
messageList = append(messageList, Message{
ID: m["id"].(gocql.UUID),
UserID: user_id,
Message: m["message"].(string),
})
userList = append(userList, user_id)
m = map[string]interface{}{}
}
}
Now we’ll take that list of user_id UUIDs from above, turn that into our map of UUID->”firstname lastname” and enrich our messages. Since we cannot inject data into our old messages list, we append the now-enriched messages into a new list, and return that as our payload.
names := Users.Enrich(userList)
for _, message := range messageList {
message.UserFullName = names[message.UserID.String()]
enrichedMessages = append(enrichedMessages, message)
}
json.NewEncoder(w).Encode(MessagesResponse{Messages: enrichedMessages})
}
Final Source: Messages/*.go
Conclusion
Building a performant API isn’t hard when you’ve learned the REST principles of keeping your data payloads simple, with one responsibility, and having appropriate JSON responses for payloads. Our example API application showed you how to create a simple API in Go built on gorilla/mux and using Cassandra as a datastore. We also showed an example of using the Stream Go SDK to save and retrieve message data with Stream’s flat feeds.
If you’d like to see more of our open-source projects, you can find them at github.com/GetStream and you can go through our Getting Started tutorial to learn more about our flat feeds and other feed types!
Also published on Medium.
