Getting Started With AWS Ansible Module Development and Community Contribution

We often hear from cloud admins and developers that they're interested in giving back to Ansible and using their knowledge to benefit the community, but they don't know how to get started. Lots of folks may even already be carrying new Ansible modules or plugins in their local environments, and are looking to get them included upstream for more broad use.

Luckily, it doesn't take much to get started as an Ansible contributor. If you're already using the Ansible AWS modules, there are many ways to use your existing knowledge, skills and experience to contribute. If you need some ideas on where to contribute, take a look at the following:

Creating integration tests: Creating missing tests for modules is a great way to get started, and integration tests are just Ansible tasks!
Module porting: If you're familiar with the boto3 Python library, there's also a backlog of modules that need to be ported from boto2 to boto3.
Repository issue triage: And of course there's always open Github issues and pull requests. Testing bugs or patches and providing feedback on your use cases and experiences is very valuable.

The boto3

Starting with Ansible 2.10, the AWS modules have been migrated out of the Ansible GitHub repo and into two new Collection repositories.

The Ansible-maintained Collection, (amazon.aws) houses the modules, plugins, and module utilities that are managed by the Ansible Cloud team and are included in the downstream Red Hat Ansible Automation Platform product.

The Community Collection (community.aws) houses the modules and plugins that are supported by the Ansible community. New modules and plugins developed by the community should be proposed to community.aws. Content in this Collection that is stable and meets other acceptance criteria has the potential to be promoted and migrated into amazon.aws.

For more information about how to contribute to any of the Ansible-maintained Collections, including the AWS Collections, refer to the Contributing to Ansible-maintained Collections section on docs.ansible.com.

AWS module development basics

For starters, make sure you've read the Guidelines for Ansible Amazon AWS module development section of the Ansible Developer Guide. Some things to keep in mind:

If the module needs to poll an API and wait for a particular status to be returned before proceeding, add a waiter to the waiters.py file in the amazon.aws collection rather than writing a loop inside your module. For example, the ec2_vpc_subnet module supports a wait parameter. When true, this instructs the module to wait for the resource to be in an expected state before returning. The module code for this looks like the following:

if module.params['wait']:
    handle_waiter(conn, module, 'subnet_exists', {'SubnetIds': [subnet['id']]}, start_time)

And the corresponding waiter:

        "SubnetExists": {
            "delay": 5,
            "maxAttempts": 40,
            "operation": "DescribeSubnets",
            "acceptors": [
                {
                    "matcher": "path",
                    "expected": True,
                    "argument": "length(Subnets[]) > `0`",
                    "state": "success"
                },
                {
                    "matcher": "error",
                    "expected": "InvalidSubnetID.NotFound",
                    "state": "retry"
                },
            ]
        },

This polls the EC2 API for describe_subnets(SubnetIds=[subnet['id']]) until the list of returned Subnets is greater than zero before proceeding. If an error of InvalidSubnetID.NotFound is returned, this is an expected response and the waiter code will continue.

Use paginators when boto returns paginated results and build the result from the .build_full_result() method of the paginator, rather than writing loops.

Be sure to handle both ClientError and BotoCoreError in your except blocks.

except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
    module.fail_json_aws(e, msg="Couldn't create subnet")

All new modules should support check_mode if at all possible.

Ansible strives to provide idempotency. Sometimes though, this is inconsistent with the way that AWS services operate. Think about how users will interact with the service through Ansible tasks, and what will happen if they run the same task multiple times. What API calls will be made? What changed status will be reported by Ansible on subsequent task executions?

Whenever possible, avoid hardcoding data in modules. Sometimes it's unavoidable, but if your contribution includes a hardcoded list of instance types or a hard-coded partition, this will likely be brought up in code review - for example, arn:aws: will not match the GovCloud or China regions, and your module will not work for users in these regions. If you've already determined there's no reasonable way to avoid hard-coding something, please mention your findings in the pull request.

Module Utilities

There's a substantial collection of module_utils available for working with AWS located in the amazon.aws collection:

$ ls plugins/module_utils/
acm.py  batch.py  cloudfront_facts.py  cloud.py  core.py  direct_connect.py  ec2.py  elb_utils.py  elbv2.py  iam.py  __init__.py  rds.py  s3.py  urls.py  waf.py  waiters.py

Of particular note, module_utils/core.py contains AnsibleAWSModule(), which is the required base class for all new modules. This provides some nice helpers like client() setup, the fail_json_aws() method (which will convert boto exceptions into nice error messages and handle error message type conversion for Python2 and Python3), and the class will handle boto library import checks for you.

AWS APIs tend to use and return Camel case values, while Ansible prefers Snake case. Helpers for converting between these in are available in amazon.aws.module_utils.ec2, including ansible_dict_to_boto3_filter_list(), boto3_tag_list_to_ansible_dict(), and a number of tag and policy related functions.

Integration Tests

The AWS Collections primarily rely on functional integration tests to exercise module and plugin code by creating, modifying, and deleting resources on AWS. Test suites are located in the Collection repository that contains the module being tested. The preferred style for tests looks like a role named for the module with a test suite per module. Sometimes it makes sense to combine the tests for more than one module into a single test suite, such as when a tightly coupled service dependency exists. These will generally be named for the primary module or service being tested. For example, *_info modules may share a test with the service they provide information for. An aliases file in the root of the test directory controls various settings, including which tests are aliased to that test role.

tests/integration/targets/ecs_cluster$ ls
aliases  defaults  files  meta  tasks

tests/integration/targets/ecs_cluster$ cat aliases
cloud/aws
ecs_service_info
ecs_task
ecs_taskdefinition
ecs_taskdefinition_info
unsupported

In this case, several modules are combined into one test, because an ecs_cluster must be created before an ecs_taskdefinition can be created. There is a strong dependency here.

You may also notice that ECS is not currently supported in the Ansible CI environment. There's a few reasons that could be, but the most common one is that we don't allow unrestricted resource usage in the CI AWS account. We have to create IAM policies that allow the minimum possible access for the test coverage. Other reasons for tests being unsupported might be because the module needs resources that we don't have available in CI, such as a federated identity provider. See the CI Policies and Terminator Lambda section below for more information.

Another test suite status you might see is unstable. That means the test has been observed to have a high rate of transient failures. Common reasons include needing to wait for the resource to reach a given state before proceeding or tests taking too long to run and exceeding the test timer. These may require refactoring of module code or tests to be more stable and reliable. Unstable tests only get run when the module they cover is modified and may be retried if they fail. If you find you enjoy testing, this is a great area to get started in!

Integration tests should generally check the following tasks or functions both with and without check mode:

Resource creation
Resource creation again (idempotency)
Resource modification
Resource modification again (idempotency)
Resource deletion
Resource deletion (of a non-existent resource)

Use module_defaults for credentials when creating your integration test task file, rather than duplicating these parameters for every task. Values specified in module_defaults can be overridden per task if you need to test how the module handles bad credentials, missing region parameters, etc.

- name: set connection information for aws modules and run tasks
  module_defaults:
    group/aws:
      aws_access_key: "{{ aws_access_key }}"
      aws_secret_key: "{{ aws_secret_key }}"
      security_token: "{{ security_token | default(omit) }}"
      region: "{{ aws_region }}"

  block:

  - name: Test Handling of Bad Region
    ec2_instance:
    region: "us-nonexistent-7"
      ... params …

  - name: Do Something
    ec2_instance:
      ... params ...

  - name: Do Something Else
    ec2_instance:
      ... params ...

Integration tests should make use of blocks with test tasks in one or more blocks and a final always: block that deletes all resources created by the tests.

Unit Tests

While most modules are tested with integration tests, sometimes this is just not feasible. An example is when testing AWS Direct Connect. The community.aws.aws_direct_connect* modules can be used to establish a network transit link between AWS and a private data center. This is not a task that can be done simply or repeatedly in a CI test system. For modules that cannot practically be integration tested, we do require unit tests for inclusion into any AWS Ansible Collection. The placebo Python library provides a nice mechanism for recording and mocking boto3 API responses and is preferred to writing and maintaining AWS fixtures when possible.

CI Policies and Terminator Lambda

The Ansible AWS CI environment has safeguards and specific tooling to ensure resources are properly restricted, and that test resources are cleaned up in a reasonable amount of time. These tools live in the aws-terminator repository. There are three main sections of this repository to be aware of:

The aws/policy/ directory
The aws/terminator/ directory
The hacking/ directory

The aws/policy/ directory contains IAM policies used by the Ansible CI service. We generally attempt to define the minimum AWS IAM Actions and Resources necessary to execute comprehensive integration test coverage. For example, rather than enabling ec2:*, we have multiple statement IDs, Sids that specify different actions for different resource specifications.

We permit ec2:DescribeImages fairly broadly in the region our CI runs in:

  Resource:
      - "*"
    Condition:
      StringEquals:
        ec2:Region:
          - '{{ aws_region }}'

But are more restrictive on which instance types can be started or run via CI:

- Sid: AllowEc2RunInstancesInstanceType
    Effect: Allow
    Action:
      - ec2:RunInstances
      - ec2:StartInstances
    Resource:
      - arn:aws:ec2:us-east-1:{{ aws_account_id }}:instance/*
    Condition:
      StringEquals:
        ec2:InstanceType:
          - t2.nano
          - t2.micro
          - t3.nano
          - t3.micro
          - m1.large  # lowest cost instance type with EBS optimization supported

The aws/terminator/ directory contains the terminator application, which we deploy to AWS Lambda. This acts as a cleanup service in the event that any CI job fails to remove resources that it creates. Information about writing a new terminator class can be found in the terminator's README.

The hacking/ directory contains a playbook and two sets of policies that are intended for contributors to use with their own AWS accounts. The aws_config/setup-iam.yml playbook creates IAM policies and associates them with two iam_groups. These groups can then be associated with your own appropriate user:

ansible-integration-ci: This group mirrors the permissions used by the the AWS collections CI
ansible-integration-unsupported: The group assigns additional permissions on top of the 'CI' permissions required to run the 'unsupported' tests

Usage information to deploy these groups and policies to your AWS user is documented in the setup-iam.yml playbook.

Testing Locally

You've now written your code and your test cases, but you'd like to run your tests locally before pushing to GitHub and sending the change through CI. Great! You'll need credentials for an AWS account and a few setup steps.

Ansible includes a CLI utility to run integration tests. You can either set up a boto profile in your environment or use a credentials config file to authenticate to AWS. A sample config file is provided by the ansible-test application included with Ansible. Copy this file to tests/integration/cloud-config-aws.ini in your local checkout of the collection repository and fill in your AWS account details for @ACCESS_KEY, @SECRET_KEY, @SECURITY_TOKEN, @REGION.

NOTE: Both AWS Collection repositories have a tests/.gitignore file that will ignore this file path when checking in code, but you should always be vigilant when storing AWS credentials to disk or in a repository directory.

If you already have Ansible installed on your local machine, ansible-test should already be in your PATH. If not, you can run it from a local checkout of the Ansible project.

git clone https://github.com/ansible/ansible.git
cd ansible/
source ansible/hacking/env-setup

You will also need to ensure that any Collection dependencies are installed and accessible in your COLLECTIONS_PATHS. Collection dependencies are listed in the tests/requirements.yml file in the Collection and can be installed with the ansible-galaxy collection install command.

You can now run integration tests from the Collection repository:

cd ~/src/collections/ansible_collections/amazon/aws
ansible-test integration ec2_group

Tests that are unstable or unsupported will not be executed by default. To run these types of tests, there are additional flags you can pass to ansible-test:

ansible-test integration ec2_group --allow-unstable  --allow-unsupported

If you prefer to run the tests in a container, there is a default test image that ansible-test can automatically retrieve and run that contains the necessary Python libraries for AWS tests. This can be pulled and run by providing the --docker flag. (Docker must already be installed and configured on your local system.)

ansible-test integration ec2_group --allow-unstable  --allow-unsupported --docker

The test container image ships with all Ansible-supported versions of Python. To specify a particular Python version, such as 3.7, test with:

ansible-test integration ec2_group --allow-unstable  --allow-unsupported --docker --python 3.7

NOTE: Integration tests will create real resources in the specified AWS account subject to AWS pricing for the resource and region. Existing tests should make every effort to remove resources at the end of the test run, but make sure to check that all created resources are successfully deleted after executing a test suite to prevent billing surprises. This is especially recommended when developing new test suites or adding new resources not already covered by the test's always cleanup block.

NOTE: Be cautious when working with IAM, security groups, and other access controls that have the potential to expose AWS account access or resources.

Submitting a Change

When your change is ready to submit, open a pull request (PR) in the GitHub repository for the appropriate AWS Collection. Shippable CI will automatically run tests and report the results back to the PR. If your change is for a new module or tests new AWS resources or actions, you may see permissions failures in the test. In that case, you will also need to open a PR in the mattclay/aws-terminator repository to add IAM permissions and possibly a Terminator class to support testing the new functionality, as described in the CI Policies and Terminator Lambda section of this post. Members of the Ansible AWS community will triage and review your contribution, and provide any feedback they have on the submission.

Next Steps and Resources

Contributing to open source projects can be daunting at first, but hopefully this blog post provides a good technical resource on how to contribute to the AWS Ansible Collections. If you need assistance with your contribution along the way, you can find the Ansible AWS community on Freenode IRC in channel #ansible-aws.

Congratulations and welcome, you are now a contributor to the Ansible project!