Supercharge Your Go Tests Using Fake HTTP Services

Why Testing With Fakes Is Important

Testing is a critical part of building reliable and maintainable Go applications. When your code interacts with external HTTP services, relying on real HTTP requests during tests can lead to flaky and inconsistent results. Network issues, rate limits, or changes in external APIs can all cause your tests to fail unpredictably. By using fake HTTP services, you can simulate real-world scenarios in a controlled environment, ensuring your tests are fast, reliable, and repeatable. In this article, we’ll explore how to set up fake HTTP services in Go to improve the confidence you have in your systems and streamline your testing process.

Configurable HTTP Clients with Environment Variables

In order for this approach to work however, you do need to ensure that you build your systems in such a way that it’s easy to specify where the clients are ultimately sending your HTTP requests to. One effective approach is to use environment variables to configure the base URLs your clients interact with. This ensures that your application can seamlessly switch between production, staging, and testing environments without requiring code changes.

Another approach is to have a sensible default base URL specified as a constant within the client code. You could then leverage something like the functional options parameter pattern to allow for the overriding of this value.

The important thing is that you have some level of control over this value within your client. This allows you to leverage the concept of fakes far easier in your test fixtures.

Here’s an example of how you can achieve this:

package client

import (
    "net/http"
    "os"
)

type APIClient struct {
    BaseURL    string
    HTTPClient *http.Client
}

func NewAPIClient() *APIClient {
    baseURL := os.Getenv("API_BASE_URL")
    if baseURL == "" {
        baseURL = "https://default-production-url.com" // Fallback to a default URL
    }

    return &APIClient{
        BaseURL:    baseURL,
        HTTPClient: &http.Client{},
    }
}

Benefits of This Approach

1. Environment-Specific Configuration: By using environment variables, you can easily configure your application to use different base URLs for production, staging, or local testing environments.
2. Simplified Testing: During testing, you can set the API_BASE_URL to point to a fake HTTP service or mock server, allowing you to simulate various scenarios without relying on external services.
3. Improved Security: Sensitive URLs or credentials can be managed securely using environment variables, reducing the risk of hardcoding them into your source code.

Example Usage in Tests

When writing tests, you can override the API_BASE_URL environment variable to point to your fake HTTP service:

package client_test

import (
    "net/http"
    "net/http/httptest"
    "os"
    "testing"

    "your_project/client"
)

func TestAPIClient(t *testing.T) {
    // Set the API_BASE_URL to the fake server's URL
    os.Setenv("API_BASE_URL", "http://localhost:10000")

    apiClient := client.NewAPIClient()
    // Perform your tests using the apiClient
    // ...
}

By designing your HTTP clients with configurability in mind, you can ensure that your application is both flexible and testable, leading to more robust and maintainable code.

Writing HTTP Tests Using httptest

Now, traditionally, a Go developer would lean on the net/http/httptest package for creating and managing test HTTP servers. These servers allow you to simulate HTTP responses and test how your application interacts with external services in a controlled environment.

Example: How to test Go Clients with httptest

Suppose you have an HTTP client that fetches data from an external API. You can use httptest to simulate the API’s behavior and test your client’s functionality.

package client_test

import (
    "io/ioutil"
    "net/http"
    "net/http/httptest"
    "testing"

    "your_project/client"
)

func TestAPIClient_GetData(t *testing.T) {
    // Create a fake HTTP server
    fakeServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Simulate a successful response
        w.WriteHeader(http.StatusOK)
        w.Write([]byte(`{"message": "Hello, World!"}`))
    }))
    defer fakeServer.Close()

    // Override the API base URL to point to the fake server
    apiClient := &client.APIClient{
        // the alternative approach is to be able to pass it in directly
        // like so:
        BaseURL:    fakeServer.URL,
        HTTPClient: &http.Client{},
    }

    // Call the method you want to test
    response, err := apiClient.GetData()
    if err != nil {
        t.Fatalf("expected no error, got %v", err)
    }

    // Validate the response
    expected := `{"message": "Hello, World!"}`
    if response != expected {
        t.Errorf("expected %s, got %s", expected, response)
    }
}

Whilst this approach works, I think it could be improved upon to provide a better experience for developers maintaining and extending these tests.

Over the past year, I’ve been working through improving the developer experience of writing a suite of fairly comprehensive and complex acceptance tests that exercise the integrations.

This has led me to develop out this library - https://github.com/elliotforbes/fakes - which makes the task of setting up one or more incredibly simple.

downstreamAPI := fakes.New()
downstreamAPI.Endpoint(&fakes.Endpoint{
    Path: "/some/path/my/app/hits",
    Response: `{"status": "great success"}`,
})
downstreamAPI.Run(t)
 
fmt.Println(downstreamAPI.BaseURL)

From my totally un-biased perspective, this strikes me as a far easier setup of a fake that allows me to really focus my attention on the logic contained within my clients as opposed to worry too much about the fake setup itself.

Let’s throw this into our earlier example to see how this plays out:

package client_test

import (
    "io/ioutil"
    "net/http"
    "net/http/httptest"
    "testing"

    "your_project/client"
)

func TestAPIClient_GetData(t *testing.T) {
    downstreamAPI := fakes.New()
    downstreamAPI.Endpoint(&fakes.Endpoint{
        Path: "/some/path/my/app/hits",
        Response: `{"status": "great success"}`,
    })
    downstreamAPI.Run(t)

    // Override the API base URL to point to the fake server
    apiClient := &client.APIClient{
        BaseURL:    downstreamAPI.BaseURL,
        HTTPClient: &http.Client{},
    }

    // Call the method you want to test
    response, err := apiClient.GetData()
    if err != nil {
        t.Fatalf("expected no error, got %v", err)
    }

    // Validate the response
    expected := `{"status": "great success"}`
    if response != expected {
        t.Errorf("expected %s, got %s", expected, response)
    }
}

This sets up a series of sensible defaults that you’d typically expect of an API, like setting the Content-Type to application/json. It also does some basic assertions that these endpoints defined on the fake have actually been called at least once.

For those of you that require a stronger suite of assertions on both how your client methods handle the responses, as well as guaranteeing that you’ve sent the correct request shape to the downstream API. You can take advantage of the Expectation field when registering an &fakes.Endpoint struct on the downstream API.

downstreamAPI := fakes.New()
downstreamAPI.Endpoint(&fakes.Endpoint{
    Path:     "/:id",
    Response: `{"status": "great success"}`,
    Expectation: func(r *http.Request) {
        // assert we've sent the correct value for the 'id' path param:
        id := strings.TrimPrefix(r.URL.Path, "/")
        assert.Equal(t, "some-id", id)
    },
})
downstreamAPI.Run(t)

As a general rule, I’d highly recommend fleshing out assertions that validate you are indeed sending the right shape of data that a downstream API expects. Perhaps validating the headers being sent with the request may be a good idea.

Simulating Errors in your Fakes

It’d be lovely if we only ever had to worry about happy-path testing and could disregard any potential error cases or sad paths. Our software systems would be much simpler and take far less time to build. However, as we live in the real world, we know that this isn’t the case and our jobs as engineers is to ensure we’re able to handle the inevitable failures.

It’s important that regardless of what testing approach you leverage, you should be writing tests that ensure you are indeed exercising these sad-path cases.

The fakes package enables us to simulate failure cases within our tests with ease:

downstreamAPI := fakes.New()
downstreamAPI.Endpoint(&fakes.Endpoint{
    Path:     "/:id",
    StatusCode: http.StatusNotFound, 
    Response: `{"error": "oh no, I've left the oven on"}`,
    Expectation: func(r *http.Request) {
        // assert we've sent the correct value for the 'id' path param:
        id := strings.TrimPrefix(r.URL.Path, "/")
        assert.Equal(t, "some-id", id)
    },
})
downstreamAPI.Run(t)

Now, when our test goes to hit our downstream API, it’ll be greeted with a lovely 404 status code and an error message that will allow us to exercise our code’s ability to handle failure gracefully.

Conclusion

In this article, we’ve explored how to set up tests for Go applications using fake HTTP services, leveraging both the httptest package and the fakes library. By adopting these techniques, you can create robust, reliable, and maintainable tests that simulate real-world scenarios while avoiding the pitfalls of flaky external dependencies.

The fakes library, in particular, aims to simplify the process of creating and managing fake HTTP services, providing a more developer-friendly experience. Whether you’re testing happy paths or simulating error cases, this library can help streamline your testing workflow and improve the overall quality of your codebase.

If you found this article helpful or have feedback on the fakes library, I’d love to hear from you! Feel free to reach out or contribute to the project on GitHub. Your input is invaluable in making this tool even better for the Go community.