Managing Kong Configuration as Code
Kong Native Solution
Kong has its own native implementation of declarative config management. This is called db-less mode and as the name implies there is no database, instead the config is stored in a yaml file and is loaded into memory at start up and can also be refreshed.
The yaml config is loaded into memory on each kong server individually so there is scope for having the config out of sync between kong servers.
Not all plugins are supported on kong running in db-less mode, the most significant for us is the Oauth plugin, which is currently being used extensively, requires a db.
https://docs.konghq.com/1.3.x/db-less-and-declarative-config/
Open Source Tools
For the POC I used decK to manage the configuration. There are other utilities eg. kongfig and Maples7/kong-config-manager. But all the tools I looked at suffered from the same issue in that they do not directly integrate with vault and we don’t want to store kong secrets in gitlab.
To get around this we will need to store secrets as vault references in the gitlab source and run a pre-processing script to resolve these before applying the configuration.
There is a vault plugin in Kong, however this plugin is only available with the enterprise edition and more crucially it is only used for managing access and secret keys, not other kinds of secrets.
Testing Locally using Docker
There is a docker-compose file in the docker-kong github repository, but when I tried this there were issues connecting to postgres and it didn’t start successfully.
While searching for a solution I found another docker-compose file that runs, kong, konga and postgres, with a little tweaking this is what was used.
Once kong was started locally and konga was available, a snapshot of the integration-dev kong was exported using konga, which was then be imported into the local kong.
Initially the import failed this was resolved by increasing the max connections to 200 in postgres.
Having imported the config from integration-dev kong into local kong I used decK to manage the config.
decK
https://docs.konghq.com/hub/kong-inc/deck/
decK helps manage Kong’s configuration in a declarative fashion. It can sync configuration to a running Kong cluster, diff configuration to detect any drift or manual changes and backup your Kong’s configuration. It also can manage Kong’s configuration in a distributed way using tags, helping you split Kong’s configuration across various teams.
Main decK functions
ping - test decK can reach kong using admin API.
dump - extracts the current config from Kong and dumps to file called kong.yaml
diff - compares the local config with the kong database config. Produces a report listing the changes that would be applied by sync.
sync - applies local config to kong
reset - goodbye kong config
Local deck Configuration file
The configuration file produced by the dump command is kong.yaml but this can be split into multiple files to make the config more manageable. For the POC I created one file per service, a file for all the consumers and another file for all global-plugins.
The diff and sync commands both use kong.yaml as the local repository by default but this can be overridden by specifying parameter --state dirname/
eg. deck diff --state deck-config/
In decK a service which contained the service itself, associated routes and plugins.
The admin URL that decK uses defaults to to 127.0.0.1:8001 this can be changed using environment variables.
export DECK_KONG_ADDR=https://127.0.0.1:8444
export DECK_HEADERS=kong-admin-token:<insert token here>
Bringing the Config under decK control
There is initial effort
Dump the kong config to kong.yaml using deck dump command
Using a simple bash script split kong.yaml into multiple service files output to a directory called deck-config/
run deck diff --state deck-config/ to confirm that the split config is good
Create another directory called gitlab-source and copy the contents of deck-config to it. This will be where the secrets are replaced with vault references.
For each file in gitlab-source check for the presence of secrets. Locate the corresponding secret in vault (easier said than done) and replace the secret with a vault reference.
Run bash preprocessing script that reads gitlab-source directory replacing vault references and output to deck-config.
Run deck diff --state deck-config/ to confirm the local config is still in sync with kong config.
When all secrets have been replaced with references push to gitlab.
Storing Secrets in Vault
References to secrets are in the same format as used previously for secrets stored in task definitions.
vault://service-pods/service-pod-1/development/development/some-squad/some-api/users/some-user[secret_key]
The kong attributes being managed as secrets are:
|
|
|
|---|---|---|
aws_key: | services/plugins | lambda plugins |
aws_secret: | services/plugins | lambda plugins |
client_id: | consumers | oauth credentials |
client_secret: | consumers | oauth credentials |
key: | consumers | keyauth credentials |
name: | consumers | oauth credentials |
password: | consumers | basic auth credentials |
provision_key: | services/plugins | oauth plugin |
secret: | consumers | jwt plugin |
username: | consumers | basic auth credentials |
Making configuration changes
Configuration stored in gitlab repository. Separate repositories for Development and Production Kong.
Secrets stored as vault references in the source prior to pushing to gitlab. Can this be enforced? For testing purposes a modified version of the vault secrets retrieval script was used
Clone/Pull repo from gitlab to local git repo
Make configuration changes in git repo directory
Run preprocess script to resolve vault references and output the config to directory deck-config/
run deck diff to check changes that will be added and confirm there has been no drift in kong config.
Commit and push local git repo back to gitlab.
Jenkins pipeline
step 1 job clones the repo runs the preprocess script to create deck-config/
step 2 job runs deck diff
step 3 manual release runs deck sync.
Jenkins requires access to all areas of vault where Kong secrets are kept, these are currently scattered around.
Jenkins will require access to kong admin port 8444 using management URL https://kong.management.integration-dev.digital.xxxxx.co.uk:8444
Users should only have browse access in konga, with admin kept for emergencies.
Nightly Jenkins Drift Check
Run to catch any manual changes
step 1 job clones the repo runs the preprocess script to create deck-config/
step 2 job runs deck diff
step 3 Conditional email report / alert that there is config drift.
Comments